Bot-X: An AI-based virtual assistant for intelligent manufacturing

1. Introduction

In the context of Industry 4.0, intelligent manufacturing plays a key role in monitoring and controlling the manufacturing processes, diagnosing the system runtime status, making intelligent decisions through real-time communication with the sensor, device, and human operator [1, 2, 3, 4]. As the most flexible entity in the manufacturing systems, operators need to handle a versatile range of tasks, e.g., checking the production process, identifying the defects of the product, reconfiguring production systems, making decisions, and so on. It is crucial to design flexible and adaptable interfaces – human-machine interface (HMI) to enhance the interaction between machines and operators for assisting operators in solving problems and making decisions quickly and easily [5, 6].

In general, there are two main challenges in designing a flexible and adaptable HMI. First, creating an HMI that works on different tablets or mobile devices is a challenge. With the development of IT technology, the high performance with low price portable devices, e.g., tablet, mobile phone, as handheld HMI devices are getting popular in manufacturing. However, due to compatibility issues, not all handheld devices work the same way. It is hard to design a unified HMI which is suitable for various handheld devices. Second, supporting an efficient and natural communication way. The mainstream of HMI is rooted in physical interaction, e.g., keyboard, touch screen, control panel. Voice commands enable HMI is still a new area to be explored.

In this paper, we propose a virtual assistant, Bot-X, to support goal-oriented communication through natural dialogues. The first phase of building Bot-X is to outline the knowledge boundary. Most of the knowledge of manufacturing is coming from three aspects, domain expert (can also be experienced operators/managers), industry standards, and work documents. In the second phase, we propose the framework of the Bot-X and list key components that enable Bot-X. The third phase is by presenting our model architecture of two layers neural networks with Softmax regression to identify the human intents. For simplicity and efficiency, Bag-of-Words (BOW) [7] is selected to implement the feature extraction for sentence generated from human speech.

To evaluate the performance of the proposed Bot-X, we integrated it with Aalborg University Manufacturing Execution Systems (AAU-MES) and an ERP system based on an open-source platform, Odoo [8]. Experimental results demonstrate that Bot-X creates a more user-friendly and efficient production environment for both manager and shop floor operators. Figure 1 describes the system overview of Bot-X based on the above three phases.

OPEN IN VIEWER

There are three main contributions of this work. 1) We proposed a virtual assistant, Bot-X, for building future intelligent manufacturing. Unlike most previous works that focus on chatbots, we pay more attention to introduce natural dialogues phenome instead. 2) We collaborated with experts who come from the manufacturing area to design the domain-specific knowledge database for Bot-X. 3) We developed a deep learning model to identify human intents from speech, which can learn the key features from the natural conversation to accurately predict the human intents.

The rest of this paper is organized as follows. Section 2 presents related work. Section 3 briefly introduces the motivation of the Bot-X and corresponding to the manufacturing cases we are currently focusing on. It also describes the Bot-X framework, architectures, and core learning algorithm. Three scenarios are defined to test Bot-X in Section 4. Section 5 discusses the benefit and limitation of the Bot-X. Section 6 concludes the paper and proposes some lines of research for future work.

In recent years, Robotics Process Automation (RPA) tools received increasing attention regarding achieving the automatic process of manufacturing tasks, e.g., Blueprism [9], Automation Anywhere [10], UiPath [11], for the configuration of task automation. Though such tools enable the rapid development and deployment of automated processes and manufacturing tasks, they still tightly depend on the running environment, e.g., operation system, programming language, device type.

The other research stream of leveraging virtual assistants can be traced back to 1966, a well-known chatbot, ELIZA [12]. It is the most famous chatbot in the history which follows rule-based architecture and able to simulate to provide sophisticated therapeutic advice. In 1971, PARRY [13], as the first known chatbot to pass the Turing test, is also developed under the rule-based architecture with a focus on clinical psychology. Though rule-based chatbots provide an efficient and natural HMI, which is also easy to be customized and controlled, it still has two limitations. They are using hand-built rules, which takes a lot of time. Moreover, it is hard to pre-define all the rules before deploying the chatbot, especially for open-domain scenarios.

On the other hand, corpus-based chatbots leverage human to human conversations to learn the pattern of communication. Such chatbots are trained on billions of words, e.g., Twitter [14], Reddit, movie dialogue [15]. Therefore, corpus-based chatbots can provide a humanlike response without hand-built rules. However, it does not consider the conversation context because it only uses the immediately previous user utterance as input. To overcome this issue, many approaches leverage the reinforcement learning to learn how to generate the appropriate responses which fit into the whole conversation context [16, 17].

To enhance the automation communication with human operators, many companies start to introduce chatbots [18, 19, 20] to the shop floor to handle tasks of different scenarios, e.g., information querying, system diagnosis, production control. Nevertheless, the major problem of chatbots is time-consuming and failed to handle natural dialogues through voice control. Due to the text-based characteristic, a conversation between an operator and a chatbot has to go through the typing. An average professional typist usually types at speeds of 50 to 80 words per minute, while the speeds of speech can reach 150–160 per minute [21, 22]. A chatbot is programmed to deal with structure dialogue and unable to understand slangs used in everyday natural conversations and analyze the sentiments of sentences. Therefore, several voice-enabled virtual assistants, e.g., Alex from Amazon [23], Google Assistant [24], Cortana [25] from Microsoft, and Siri [26] from Apple, are developed and widely used in the context of entertainment or personal-oriented [27]. The key feature of those virtual assistants is they have a robust natural language process (NLP) capacities and able to handle continuous natural dialogues [28, 29].

3. Proposed Bot-X

In this section, we firstly present the motivation behind the Bot-X development through manufacturing cases based Smart Production Line at Aalborg University [30]. It answers four research questions, 1) what’s the background of building such advanced HMI, 2) how to define the knowledge boundary, 3) what values it can create for the manufacturing domain, and 4) how can we promote it to different industrial cases. Then, we describe the design principle of Bot-X.

4. Case study

Manufacturing environments are typically highly organized, structured, controlled, and rule-based. The correlated business operations and processes are normally located upstream. Salespersons and shop floor workers/robots do not share the same working space. Therefore, it is important to have a unified platform that can share or monitor the whole processes for the different types of users in a manufacturing company. As a testbed scenario, we chose an open-source business platform – Odoo, which provides modules and interface for both business and manufacturing purposes. Thus, by implementing the Bot-X into this platform, the entire production processes, i.e., sales order generation, manufacturing order creation, and production execution, can be integrated together as an integration solution for small and medium-sized enterprises (SMEs) to build a more efficient and productive intelligent manufacturing factory. An example of the current work environments is shown in Fig. 8.

The Bot-X is tested in different scenarios where the focuses were on assisted sales order generation, assisted manufacturing order creation, and assisted production control. The demo is available online.1

In the first scenario, Bot-X extracts the sales information from the natural conversation with the user. Such information consists of the customer name, product name, and product quantity. Usually, those information is manually entered into the ERP system by operators. To this end, an operator initials a dialog with a human user to command the Bot-X to perform a task, e.g., produce 6 flowers (the flower is made by Lego bricks) for John. The Bot-X uses the convolutional neural network (CNN) model to predict the human intent and infer the named entities from the user utterance. The clarification dialog is used to update the confidence from a user response if Bot-X failed to identify the entity from user utterance. Table 2 gives some examples of the policy of clarification dialog. Based on the confirmation question responses, the Bot-X stores the identified entities and maps them to the input parameters of the selenium API calls for sales order generation. A conversation concludes when the operator has confirmed the entities. Then, Bot-X invokes the selenium API to open the ERP system and automatically perform the sales order generation process, i.e., searching and choosing the customer and product from the system and assigning right quantity number to the sales order.

OPEN IN VIEWER

The second scenario is related to the task of creating a manufacturing order: the Bot-X checking the inventory to verify if the materials are enough to produce the product. This task is often carried out in the warehouse: once the sales order is specified, it will be automatically sent to the warehouse. The employee of the warehouse needs to check the bill of materials (BOM) from the database or inventory system. The manufacturing order will be created if the reserved quantity of required materials is enough to perform the production. Though such task does not require high skills worker, massive amount of BOM confirming and verifying by a human operator might lead to low efficiency and accuracy. Therefore, Bot-X can easily perform the above task preciously with high reliability. After receiving the voice commands, the Bot-X can directly invoke the selenium API to tracking the materials in the inventory system and verify if the stored materials meet the requirement of the sales order. The clarification dialog is also used to confirm human intent. Table 3 shows some examples of the clarification dialog.

The third scenario we tested is production control. Normally, production execution is started by shop floor operators. This task might require an operator to manually manipulate the physical switch or click the button on a touch screen through an HMI. Though some physical operations have to be done manually, other operations (e.g., sending the commands to the PLC of a production module) may be able to perform through a virtual channel. Bot-X is then expected to retrieve the information of manufacturing order, production scheduling, and equipment status from the AAU-MES. In our case, Bot-X plays a role as a middle agent between the production module and Odoo ERP/AAU-MES system. It is invoked by the user to ask a simple command, e.g., please start the production now. After identifying the human intent, the Bot-X will start to check the production schedule from the AAU-MES to see if any production is planned and needs to be executed. The Bot-X will use clarification dialog to confirm with the operator if the production is ready or not, i.e., the planned production is found and all the production modules are ready to work. The programmable logic controller (PLC) of production modules will start to control the production module to perform the production after receiving the commands and production parameters from Bot-X. Table 4 shows some examples of the dialog policy of production control.

To overcome the shortcoming of RPA, we developed Bot-X to support different platforms with a smooth learning curve. Python, which is in use at many places as an integration language to glue existing components are written in other languages (e.g., C/C++, Java), is chosen as a back-end programming language in this work. Thus, the “brain” of the Bot-X can be deployed on any server regardless of the hardware/software compatibility issues. The handheld devices only need to provide a user-friendly front-end interface that receives and transfers operators’ requests and leaves the computation part to the back-end server. The front-end interface is enabled through Bot-X combing with manufacturing domain knowledge to facilitate human-machine communication for building intelligent manufacturing.

Through the natural dialogue, the Bot-X understands the user utterance, which is grounded in the manufacturing environment, and natural language systems can help to extract valuable information for later operations based on the semantic analysis. However, there is a number of challenges that still need to be addressed before we deploy the virtual assistant in the production line, for example, ambient noise. The above experiments show that the microphone picked up a lot of ambient noise during the conversation. The accuracy of identifying the entities from user utterance is influenced by the ambient noise. Currently, Bot-X sets the dynamic energy threshold to automatic. It helps to adjust the microphone to filter the ambient noise automatically. However, the adjustment process is also time-consuming, which may lead to low efficiency.

6. Conclusions

In this work, a voice-controlled virtual assistant, Bot-X, is developed and applied to manufacturing environments. In particular, the Bot-X is tested in three scenarios to demonstrate the flexibility and easy of use by introducing natural language interaction between humans and machines.

Different from the entertainment domain, the manufacturing environment mainly focuses on the limited, fixed, and atomic actions, e.g., power on the machine, pick up the materials, create a sales order. Introducing higher levels of interactive dialogue leads to reduce the complexity of operation systems (e.g., ERP system, production system), reduce repetitive operations, and improve productivity. In particular, the most important aspect of this kind of interaction is the virtual assistant learned to understand human intents and executed commands without human intervene.

Future works will focus on how to reduce the ambient noise influence on Bot-X. Besides, several interviews are planned to collect feedback from our industrial partners regarding design user-friendly dialogue templates.