Getting AI Implementation Right: I nsights from a G lobal S urvey

What We Do and Do Not Know about AI Implementation

Propelled by advances in computational power, programming science, and access to large data sets, ⁸ AI technologies are reaching a crucial stage of development ⁹ in areas such as machine learning, pattern recognition, computer vision, and natural language processing, ¹⁰ to name but a few. AI is commonly defined as “the ability of a machine to perform cognitive functions that we associate with human minds, such as perceiving, reasoning, learning, interacting with the environment, problem-solving, decision-making, and even demonstrating creativity.” ¹¹ Like other information technologies, AI will transform work by taking over tasks previously carried out by humans, allowing humans to focus on more complicated and rewarding tasks. Unlike most information technologies, however, AI can augment human judgment and blend with human activities in entirely new ways and diverse settings (e.g., human-AI surgery and semi-autonomous drones). ¹² AI solutions are already changing work, transforming expertise, reshaping occupational boundaries, and introducing new methods of control and decision-making. These developments fuel predictions that investments in AI will continue to increase and that productivity increases will subsequently follow. ¹³

According to scholars and thought leaders, however, firms need to rethink how they organize and operate in order to reap these benefits from AI. For example, firms must change their operating model from one in which they deliver a specific product, service, or solution to one in which they design a “software-automated, algorithm-driven digital ‘organization.’” ¹⁴ Work practices must be re-engineered in this transformation as workflows are broken down into smaller tasks corresponding to individual AI algorithms’ relatively limited capabilities. ¹⁵ Firms will also need to put in place new roles (AI-business translator, data scientist) and invest in setting up new units, such as “AI factories” with data pipelines, algorithm development, experimentation platforms, and related software architectures. ¹⁶

In this new landscape, where AI as a rapidly evolving general-purpose technology will continue to find new application areas, managers are likely to become less decision-makers and more curators of portfolios of algorithms and data flows, and employees that possess both AI and domain-specific skills will be particularly sought-after. ¹⁷ In parallel, concerns regarding the liability, trustworthiness, and ethical usage of AI algorithms, including risks for privacy violations through AI and data, are being raised, ¹⁸ alongside proposals for regulations to address how and where AI should be implemented. ¹⁹

The many distinct characteristics of AI technologies make it essential to consider the AI implementation challenge as different from implementing established information technologies, such as enterprise resource planning (ERP) systems, customer relationship management (CRM) systems, HR systems, and productivity and communication software. These are typically stable and well-integrated software products with highly controlled (and relatively infrequent) release schedules and with high reliability and predictability in their functionality. This contrasts with the often granular, dynamic, tentative, and incomplete functioning of combinations of AI algorithms, which exhibit much greater volatility in evolution and use. Indeed, the potential, use, consequences, and expertise needed for AI are quite different from traditional IT, and it thus becomes essential to understand how AI can be implemented. That is, we must figure out how to design, configure, and deploy AI technologies and adapt organizational structures and routines to realize the technology’s potential while accommodating different demands. ²⁰ However, literature on AI implementation in organizations is scarce. A few studies have highlighted the importance of AI-relevant competencies (e.g., data scientists with deployment-oriented skills or domain experts that can make productive use of data) ²¹ and stressed risks arising from the underperformance of AI technologies ²² and the lacking availability and quality of data. ²³ While these and other studies highlight the usefulness of attending to technological as well as organizational and cultural (people) challenges, ²⁴ much remains to explore to provide actionable knowledge for managers charged with leading AI implementation initiatives. We thus set out to survey a large sample of firms to discover specific challenges and solutions associated with AI implementation.

Data and Research Design

We combined expert interviews with a survey of white-collar decision-makers. As a first step, we conducted 16 in-depth interviews with AI implementation experts in Sweden and the United Kingdom in sectors such as IT, telecommunications, banking and finance, insurance, fintech, and the public sector. The experts represented different roles, such as chief technology officers, chief digital officers, government experts, and suppliers of AI and platforms, all with extensive experience in implementing AI. The interviews revealed that implementing AI is a complex and laborious process. They confirmed the relevance of capturing challenges not only around the technology but also including a broad spectrum of organizational and cultural issues. Based on these insights, we developed an online survey to probe into the challenges of AI implementation ²⁵ in the three distinct domains: technological, organizational, and cultural.

Next, we collected survey data from 2,525 decision-makers in China, Germany, India, the United Kingdom, and the United States with experience in implementing AI. We sampled respondents from panels of online business professionals adhering to ESOMAR ²⁶ quality controls. In addition, we assessed the number of surveys taken, response patterns, number of screen-outs, and real-time digital fingerprinting against fraudulent behaviors. We divided surveys equally among countries to reach a target quota of 500 decision-makers with AI implementation experience from each country. On a per-country basis, quotas were further subdivided to survey at least 250 technical managers responsible for introducing AI technologies into their companies and 250 operational managers tasked with using AI in their operational processes. To reach these quotas, we sampled 10,024 full-time professionals in companies with at least 100 employees, of whom 6,781 were white-collar decision-makers. A mix of panel sources was used to avoid country-specific effects and biases. Each country’s targeted audience was identical, and panels had no pre-existing focus on AI that could influence the sample.

We asked respondents to share information about their organizations’ historical and current initiatives to implement AI. Survey questions addressed the challenges encountered during AI implementation, strategies to overcome those challenges, and plans for investment and hiring. We pre-tested the survey for both understandability and translation of the different languages. On average, it took respondents 18 minutes to complete the survey.

We also conducted additional analyses to distinguish two subcategories of firms—AI Experienced firms and AI Newcomers—to better understand how the challenges of AI implementation vary with experience. We defined AI Experienced firms as having at least one fully implemented AI system and AI Newcomers as actively pursuing AI solutions but not yet having completed their first AI implementation effort. ²⁷ Given the early stage of AI implementation, we opted for an inclusive understanding of being AI Experienced to capture companies ahead of the curve (note that three out of four firms initially approached were deselected due to not being active in AI implementation). This means that our study reaches well beyond the all-known “AI superstars,” such as the Big Tech firms, to capture AI challenges and solutions among the many. In the following sections, we report on commonalities and differences in implementation challenges between AI Experienced firms and AI Newcomers.

The Technological, Organizational, and Cultural Challenges of Implementing AI

Our expert interviews taught us various challenges associated with AI implementation, from lacking visibility of available data to employees preferring to trust their intuition over data analytics. Dividing the AI implementation challenges into three domains—technological, organizational, and cultural—we analyzed each challenge based on our survey data. This revealed that nearly all (99%) of our respondents had encountered at least one challenge in one of the domains as they implemented AI. Ninety-one percent had encountered challenges in all three domains. As indicated in Figure 1, AI Experienced firms and AI Newcomers face many similar challenges (see all challenges where only the grand mean is displayed). As we compared organizations based on the maturity of their AI initiatives, an overarching insight emerged: Gaining experience does not lessen the trials and tribulations of AI implementation. Instead, increased maturity comes hand-in-hand with new challenges and, in some cases, exacerbates existing ones.

OPEN IN VIEWER

Figure 1.

Technological, organizational, and cultural challenges.

Figure 1 shows the compound results for the full sample (both AI Experienced and Newcomers) along the dimensions of technological, organizational, and cultural challenges, as well as the identified statistically significant differences between the two groups of firms. ²⁸ The first pattern that emerges is that some challenges are more prevalent than others, and many of these are of a technological or cultural rather than organizational nature. The most frequently reported technological challenges are the need for dedicated software and hardware and related investments and data management issues. Indeed, many technological challenges can be traced back to data: data not being available, data not being structured for the desired use, and data definitions not being unified across the company. One of the AI implementation experts we interviewed explained that without a unified language, structured data assets, and proper coordination across the company, it becomes nearly impossible to implement AI (see Box 1).

OPEN IN VIEWER

Box 1.

AI Leadership Builds on a Unified View of Data.

When bank A (an AI Experienced firm) started its transformational journey to becoming AI-driven, different units handled data locally. The lack of coordination between units led to several challenges. For instance, each unit had developed its terminology for different information, which led to confusion when communicating across units. Another challenge was the lack of visibility on what data were available within each unit. Searching for data became time-consuming, often achieved by informally asking around within one’s network. In turn, this practice increased the chance of misunderstandings, leading to late, manual, and costly data set corrections. The bank realized that there were significant efficiency gains to be made by becoming more organized and that they would need to make data management a prerequisite if they wanted to advance in AI development. They created a unit with the sole purpose of helping the bank manage data as an asset. The unit is now in charge of the bank’s strategic development in terms of data, including developing its data governance model and delegating data ownership within the bank.

Interestingly, we found that AI Experienced firms are more, rather than less, likely to face challenges with bending AI technologies to their will, reporting a lack of fit with industry-specific needs and problems with leveraging the technology to get at salient insights. Experienced firms are also more likely to foresee the risk of breaking laws and regulations. (One of our experts pointed out that laws and regulations hamper innovation and learning, as complying with them is costly and restraining.) We detect a growing appetite for more advanced solutions as firms gather experience: Experienced firms push the envelope regarding ambitions and complexity, running into restrictions concerning technology and data that newcomers have yet to face. They make progress, but life does not get easier. For example, AI Experienced firms are more likely to have resolved issues around data ownership, partly by promoting a data-driven workflow across organizational boundaries, while facing more technology-related shortcomings.

The most frequently mentioned organizational challenge (by some margin) is the lack of adequately trained employees. Still, the lack of understanding of AI among managers, as well as among customers, also stands out. This suggests that despite the extensive investments already taking place, considerably more investment in skills development can be expected, even in experienced firms. One of our experts shared how leaders within a data-savvy firm were reluctant to accept data analytics in decision-making. The firm had a data scientist embedded within each product-development team, and one of the tasks of the data scientist was to run an A/B test for new ideas. However, when the test showed that the idea would not measure up, the data scientist faced an escalating conflict with the product owner. The product owner even preferred statistically ambiguous results since they afforded “interpretive flexibility” and allowed for acting on gut feeling.

While building skills and understanding takes time, we see that experienced firms have made some headway: Newcomers experience more challenges concerning employee skills. They also report a higher incidence of challenges related to AI and data governance (lack of clear ownership). The good news here is that effort pays off: Experienced firms are gaining ground in the form of reduced incidence of some organizational challenges. However, this is also a story of perseverance, where challenges evolve but remain as complexity grows.

Finally, companies also struggle with cultural challenges. The most common occurrences are the combination of different fears and concerns that employees carry concerning AI (job loss, loss of expertise, and autonomy), in combination with inertia in routines and a perceived generational gap in technology understanding. Generally, companies report a higher prevalence of cultural challenges than organizational ones. Moreover, newcomers report significantly more cultural challenges than experienced firms: Employees are more fearful of AI, and there is stronger resistance to the technology, sticky routines represent a more frequent hurdle, and employees fail to see what value AI can bring to the business and to job satisfaction. As one expert explained, working with organizations that lack a data-driven mindset, it is a psychological challenge to get employees to realize that their intuition, often built on many years of experience, may be wrong and that they instead need to trust the data. In contrast, experienced firms more frequently report that trade unions oppose AI adoption (although, overall, this is a lesser challenge). In sum, cultural and technology challenges are front and center as companies enter the AI arena. Still, experienced firms do a better job addressing them when new challenges come to the fore.

Solutions to Advance AI Implementation

Having established the challenges that AI Experienced firms and Newcomers experience, we now turn to the solutions that managers employ to address the challenges of AI implementation. Here also, we find many solutions to be shared for experienced firms and newcomers (see Figure 2). Among technological solutions, improving the usability of AI tools was most often mentioned, followed by improved access to data, enhancing the quality of output from algorithms, improved data management, and finding suitable tools. Note that all these solutions focus on the core work of getting algorithms to operate in alignment with expectations to produce value-adding results. Most solutions we identified were employed in roughly equal measures. Still, we detected significant differences concerning tool selection, with newcomers more inclined to prioritize standardized solutions and experienced firms more often looking for tools that provide transparency and regulatory compliance. Again, this suggests that as firms become more experienced, they are hitting new hurdles, such as algorithmic opacity (driving the need for transparency) and more advanced solutions pushing privacy and legal boundaries (driving the need to ensure regulatory compliance).

OPEN IN VIEWER

Figure 2.

Technological, organizational, and cultural solutions.

As with challenges, firms seemed to pay less attention to organizational solutions than technological and cultural solutions. Also, the spread in popularity across different organizational solutions is minor (see Supplemental Appendix 2 for details). In other words, a broad range of solutions is similarly crucial. Many of these solutions focus on organizing and governing AI innovation activities, including creating new roles, building cross-functional teams, and developing control measures such as budgets and evaluation criteria. Here, we also identify two areas where newcomers and experienced firms differ: Newcomers focus more on building cross-functional teams.

In contrast, experienced firms focus more on promoting a data-driven workflow across organizational boundaries and creating AI evangelist roles. Both these latter solutions are typical for firms that have already made headway and gained momentum. Before creating an organization-wide AI evangelist role, you need early wins to build on and people with experience to share; otherwise, the evangelist can come across as a false prophet. Similarly, data-driven workflows across organizational boundaries require involving customers and partners and need to build on a foundation of early successes and honed capabilities.

Finally, the cultural solutions are focused on skills development and change management. Three stand out: workshops and training, promoting employee skills development on AI, and changing work routines. Proof-of-concept studies, demos, and a slew of other activities are also used to push the advancement of AI. Further stressing the importance of training is that newcomers focus even more on AI skills development.

Key Insights for Managers

Invest in People. Then Invest Some More

A strong pattern across challenges and solutions, and across experienced firms and newcomers, is the need to attract and develop people. While rapid technological advances drive AI adoption, AI implementation is not a technology problem best solved by a few (or many) dedicated data scientists. It is an organizational transformation and value-creation challenge driven by technology and data solutions, people, and supporting organizational arrangements in concert. Our respondents report many cultural and organizational challenges equally important for AI to gain a foothold and garner momentum (see Figure 1). We recall that 91 percent of our informants reported challenges across all surveyed categories—technology, organization, and culture. People-related issues stand out among both cultural and organizational challenges. So, investing in AI means investing in people.

For example, consider the major European bank that used AI to become “data-driven.” The initiative took off only when they connected the technology with people and purpose (see Box 2). Similarly, the head of the analytics team at another European bank shared the insight that when people do not understand the substance of AI—what AI really entails—they gravitate toward quick fixes with unrealistic expectations of results. You cannot just hire a couple of data scientists and expect wonders. You need people who know what data can be made available to feed the AI algorithms and how to interpret and make use of the results. On the other hand, brilliant data scientists who do not understand the operational context will have a hard time creating true value. A bank representative told us that data scientists with dazzling tech skills, cutting-edge statistical acumen, and compelling academic CVs are often hampered by a lack of understanding of the business and how to create value, hence the need for broad skills development and cross-functional teams, as shown in our survey results.

OPEN IN VIEWER

Box 2.

Technological Awareness Is Not Enough.

Bank B (an AI Experienced firm) started its journey toward becoming data-driven with a top-down approach. Part of the bank’s strategy was to set up an information governance model, where appointed staff became data owners responsible for the data in the bank. Data ownership could include being responsible for storage, access, and quality assurance. The approach was partly successful at creating awareness, but it did not yield any results regarding activities or initiatives. The bank then adopted a more agile working method, focusing on smaller pilot projects. This was done by identifying and approaching units with indicators of problems related to data management, for instance, being fined by or receiving reminders from the financial regulatory authority. Together with these units, the bank created pilot projects that could also work elsewhere in the organization. This time, the employees acted quickly and did not question the purpose of the information governance model. The bank recognized these changes as an effect of the increased awareness created by the workshops in the first part of the initiative.

A striking difference between AI Experienced firms and AI Newcomers is that the former have made considerably more headway in their people skills and attitudes. AI Experienced firms less often report a shortage of AI-skilled employees, fewer challenges handling employees’ fear of AI, and a smaller generational gap regarding employee preparedness. They are also more proactive in working with unions to frame AI as an opportunity for employees rather than a threat to their livelihood. Where newcomers struggle to find the right expertise and motivate employees, experienced firms have a more refined management understanding of AI.

Overall, the focus on learning is striking: proof of concepts and demos, education/training/workshops, and encouraging personal growth all rank highly. This suggests that for sustained AI adoption and AI-driven change, companies must create an informed, interested, and engaged workforce able and willing to work routinely with AI-based process improvements and solutions development. This means that investments in AI need to be combined with dedicated investments in people who can grow and remain innovative as AI technologies evolve. Many experts highlighted the need to stimulate employees to learn about data and AI. For instance, one of our experts shared that to encourage openness to learning and counter resistance, their technology firm encourages continuous learning and promotes a growth mindset. Another of our experts argued that the point of training is not to turn everyone in the organization into data scientists but that everyone needs to understand the basic concepts of data and AI, as well as develop an understanding of data and AI that cannot be taught through a traditional digital course but must be rooted in personal experience.

Shift Your Mindset: From Software to Algorithms and from Stable to Dynamic Governance

Creating an AI-driven organization places new challenges on how to manage digital technologies. Even companies with excellent IT expertise need to adapt to the world of dynamic algorithms voraciously hungry for data. The initial implementation of AI applications to address specific challenges is a good start, but it is a far cry from creating an AI-driven organization. It is also very different from managing standard IT resources, as it needs constant and iterative attention from both technical and operational domain experts after deployment. ³⁰ Regular IT systems—think of CRM systems or ERP systems—are expected to perform reliably over long periods, supported only by planned, infrequent updates. Such standard software can be developed by vendor organizations and deployed in a similar fashion across many organizations.

In contrast, to ensure that each AI application delivers value, not only does development need to be done in close coordination with operations, but the algorithms also need constant care to ensure that they perform as intended. This highly specific oversight is needed partly because AI applications are often trained on data streams from the same settings where they will be deployed. Identifying what data to include and how those data should be interpreted calls for advanced, domain-specific knowledge of business goals, existing processes, and the context in which data are derived and used. This can also include data from other AI applications creating an intricate ecosystem of data; dynamic, reusable, and modifiable algorithms; and resulting solution bundles. As AI applications evolve, they can improve their performance but also deviate or fail, requiring adjustment or retraining. This, again, calls for cross-disciplinary expertise from both the technical and operational domains.

Having only a few AI applications, this is not necessarily a problem, but as the number of AI applications grows, so does the complexity of the algorithmic ecosystem in which they are implemented. Developing and curating algorithms will call for much hands-on work—a process that can be both time- and resource-consuming and sometimes unsustainable (see Box 3). Furthermore, having multiple, interdependent AI applications—each contributing to different processes and drawing from different data sets—puts high demands on companies to organize data in a manner that avoids conflicting decisions and processes, both as data sets are modified and added and as AI applications evolve, collaborate, and depend on one another.

OPEN IN VIEWER

Box 3.

Toward a Single Platform: Standardization Can Mitigate Escalating Maintenance Costs.

At technology firm A (AI Experienced firm), a team of data scientists supported a wider organization consisting of numerous accounts to develop AI application models for operational work. At first, the number of requests for the team’s service was modest, and little coordination was needed. However, as the number of requests grew, the team soon realized that several data scientists were working on applications for different accounts that could be merged into one. Beyond duplication of work, the greater number of applications resulted in higher maintenance costs. To address these problems, the data scientist team decided to standardize the AI applications. They appointed two team members as gatekeepers responsible for investigating all novel requests and synchronizing work where needed.

Having more AI applications in place not only increases the complexity of implementation and management of the technology but also impacts the operational domain competence. One of our experts compares the difference between AI and traditional IT by explaining that AI comes much closer to the employee and, in practice, becomes an extension of that person’s competence. Instead of supporting an employee’s activity, the AI performs part of it. And the employee needs to understand what the AI has done (and why) and needs to incorporate this knowledge into their domain expertise.

Climbing the AI implementation ladder also becomes markedly costlier when applications involve and rely on customers. As the appetite to create more value grows, companies need to consider the wider ecosystem and exploit opportunities beyond the company’s borders. Companies cannot bet on finding the necessary data only on the inside of their organization, and securing commitment outside the home organization adds complex new people challenges.

Diagnosing AI Implementation Activities on Different Levels

The aforementioned insights point toward the need to manage emerging complexities as AI implementation efforts, which often start small, become more pervasive and sprawling, ultimately reaching outside the organization. Figure 3 provides three sets of questions managers must ask as the company gains AI maturity. The first set of questions will help managers gain momentum with AI initiatives in their local setting. The second set prepares managers for the internal complexities arising when the organization’s volume, ambition level, and complexity of AI implementation initiatives increase. The third set identifies critical issues managers will likely face as curating a growing AI ecosystem of partners, algorithms, and data sources become increasingly critical.

OPEN IN VIEWER

Figure 3.

A diagnostic test for AI implementation.

Conclusion: A Journey without End

If you are an AI Newcomer, a split vision is required. Initially, it is imperative to focus on concrete and delimited use cases with clear value propositions and limited complexity in algorithm development, data access, organizational scope, and risk management. At the same time, you must also prepare to manage emerging complexities by proactively investing in data management capabilities, a more fine-tuned portfolio of AI tools, broad people involvement and skills development, deep AI expertise, and nuanced management understanding.

If you are already AI Experienced, your next-level challenge will likely involve dealing with an increasingly complex ecosystem of algorithms, data, solutions, and partners. Add changing work processes and negotiating boundaries and responsibilities with employees to this mix. Some AI Experienced firms will stumble as they navigate the increasing complexity inherent in mastering the integration of AI within the firm and across partner organizations. For more experienced firms, a key insight for navigating the challenges of AI implementation is that you will never be fully AI-proficient. Instead, new and different technological, organizational, and cultural challenges will conspire to play tricks on even the most successful organizations. From its often-disorienting beginnings to the unexpected challenges of growth and maturity, AI implementation is likely to be a journey without end.

Supplemental Material