Machine learning–based automated image processing for quality management in industrial Internet of Things

Introduction

Over time, our reliance on technology has been increasing tremendously. But there are certain areas where the relationship we have with information technology (IT) is very critical. Our dependence on technology has brought about a massive change in the way we interact with the rest of the world. Computer systems were once independent of each other, but now they are specifically being designed to interact with each other as well as with humans. Devices are being networked everywhere and for a huge range of applications. Most of these devices have built-in sensors that collect data and enable devices to interact with each other. These collected data may then be delivered to people and other devices on demand. In this article, we propose a quality management system based on industrial image processing in which the various devices that enable this are interconnected via a cyber-physical cloud system (CPCS).

In industries, products are generally mass produced. Whenever mass production is involved, it comes with various challenges: quality issues, efficiency, costs, and time consumption. To overcome these problems, many industries have resorted to automation. When it comes to maintaining the quality of the products, there has to be a system at the end of the production line that scans the end products for defects. Most industries have turned to use automated systems that apply image processing to address the above-mentioned challenges. In automated industrial production processes, an image processing system plays a crucial role. ¹ In recent times, with the availability of highly efficient modern digital cameras and communication interfaces, image processing has become faster and more productive. The quality of image processing software has become more reliable while simultaneously reducing the cost.

But automizing the industrial manufacturing system comes with its own challenges. Time and central processing unit (CPU) efficiency have become increasingly important. Ensuring quality is the top priority among manufacturers. Machines that are able to implement computer vision efficiently have to be designed such that components can be identified during production flow.

There has already been a lot of research lately in industrial vision systems. In recent times, the machine vision systems used for inspection in industries achieve higher efficiency and robustness compared to older machine vision systems. Researchers are striving to make industrial vision systems that are faster, more efficient, and cost proficient. The industrial vision systems can be used for inspection of dimension quality, surface quality, structural quality, and operational quality. ² Now, depending on this classification, the type of processes or techniques used during the inspection may vary. In this article, we are concerned with the structural quality, that is, the correct assembly of the objects in question.

Research in machine vision system for inspection of structural quality of a device has been done in areas ranging from semiconductor and electronics industries, to car assembly lines. Moganti et al. ³ discussed various algorithms and technologies for automated visual inspection of printed circuit boards (PCBs). A template matching technology for automatically detecting irregularities in a car production line was introduced in Kim et al., ⁴ which was later improved in Chung et al. ⁵ to accommodate inspection of the constitution of the car doors. Techniques for automated inspection of railroad lines were given in the literatures.^6–9

Industrial image processing is generally used in automated inspection systems. Image processing operations are used to improve the image quality, enhance desired features, and so on. Analysis of the enhanced images enables pattern recognition. Pattern recognition concerns in finding similarities and patterns in images which help in classifying the objects or components present in the image. These techniques along with machine learning are exploited to successfully detect structural defects in an object. The analysis tools used recently are based on neural networks (NNs), fuzzy logic, and genetic algorithms (GAs). ⁶

In recent times, machine learning techniques have come to be used in inspection systems. ¹⁰ This is an improvement over the previously popular template matching ¹¹ and design-rule verification ¹¹ approaches. Visual inspection systems based on machine learning generally use typical supervised learning algorithms like decision trees, ¹² statistical classifiers, ¹³ artificial NNs, ¹⁴ and support vector machines (SVMs). ¹⁵ In this article, we use multi-instance (MI) learning ¹⁶ for defect detection.

Among other applications, MI learning has been previously used for image retrieval, ¹⁷ object recognition, ¹⁸ target tracking, ¹⁹ and image categorization. ¹⁷ In Paul et al., ²⁰ the visual inspection system is modeled as an MI learning problem.

In this article, we concentrate on quality management during the manufacturing of CPUs. During the assembly of CPUs, there are a number of components that could be misplaced or go missing due to certain errors. At the end of the assembly line, there has to be a protocol in place to assure the quality of these products. This article proposes to develop a system that ensures quality, efficiency, and minimum time consumption at the end of the production line. CPUs consist of various components and each of these components has to be checked for defects. Hence, MI learning can be used as the machine learning algorithm here. Also, there has to be a competent way for all the devices involved to interact with each other efficiently.

The sensors, actuators, and other devices that are interlinked and connected to form a vast network that is generally referred to as smart networked systems and societies (SNSS). ²¹ The devices are not restricted to just stationary devices but may also include mobile devices. SNSS bridges the physical and the virtual world to form cyber-physical systems (CPSs). The main objective of CPS is to connect humans and the cyber world via sensors and actuators. CPS connects numerous devices and results in a huge unfathomable exchange of information. In this article, we use an architecture that integrates CPS and cloud computing.

The CPCS has to manage all the resources connected to the system. It contains sensors, actuators, processors, and data services. All operations have to be performed via cloud servers. The cloud servers allocate resources such as bandwidth, storage, and energy as per their demands. CPCS is a very complex system that manages, monitors, and shares information in the real world. The management and optimization of such a complex system is a very challenging task. This entire infrastructure has to be operated efficiently. Because of the number of devices and organizations attached to this, even a small error will have huge repercussions on the entire system. CPSs can assist industry in controlling the complex industrial processes via sensors, actuators, and processors. ²²

The rest of this article is arranged as follows. Section “Related work” gives the research that has been done in the area at hand and various technologies or algorithms that are relevant. In section “Proposed research,” we discuss the proposed industrial image processing–based system. The experimental results and analysis are given in section “Experimental results and analysis.” Finally, section “Conclusion” concludes this article.

Related work

Manual assessment of products that are mass produced is time consuming and costly, sometimes inaccurate and complicated. To overcome these problems of quality control and quality management, many industries have taken the assistance of automated services. Image processing takes a huge role in this automation. Various techniques have to be performed to enable automation in quality control in an industry. Generally, image processing is used to compare the defects in the products.

For detection of defects in products, missing parts or components in a product have to be first identified. Some research has been done in detecting missing parts or components in a product have to be first identified. Some research has been performed in detecting equation that was used to estimate the shape of missing regions. Rather than estimating the missing region in each image independently, a joint group action was used to obtain the complete version of the original image. ²³

In the study by Vasilic and Hocenski, ²⁴ to maintain visual inspection and quality control, an automated system for the ceramic tiles industry was proposed. Various methods for detecting defects in color and surface of ceramic tiles were discussed. Most of the industrial image processing focus on just the defects but not on the process of taking the images that have to be processed. For this, occlusion handling is critical. Zhong et al. ²⁵ proposed a method to shorten the time to locate light-emitting diode (LED) chips on wafer. They proposed a fast template matching method that used image segmentation and blob analysis to predict the positions of potential chips. Experimental results on this system showed that the algorithm was much faster than the traditional methods used to previously locate LED chips.

Occlusion handling for vehicle collision handling was proposed in Paul et al. ²⁶ In this article, we propose to use occlusion handling technique to calibrate the camera that takes images before they are processed. The remainder of this section briefly discusses the technologies required in this area of research.

Proposed research

In an industry where products are mass produced, for example, a CPU production line, a tiny defect like a missing screw may go undetected when manually processed. Also, the mere quantity of products to be processed for quality control may overwhelm the industry. In this article, we propose a system to detect abnormalities in the products.

In this system, images of the end product are taken at the end of the production line. These images are used to detect defects in product.

Modules

This system consists of basically four modules (Figure 1):

Edge detection to separate the image into various components.

Processing the various parts to detect missing parts.

Predicting the description of the missing part (region in-painting).

Correcting possible camera occlusion.

OPEN IN VIEWER

Figure 1.

Pictographic representation of the various modules.

Edge detection to separate the image into various components

We need to first scan the image and separate it into various components. The basic steps of primary edge detection are as follows:

Smoothing: This process suppresses the noise.

Edge enhancement: This step filters the image for contrast.

Edge localization: This determines which local maxima from filter output are actually edges versus noise using thresholding. In thresholding, a threshold value is chosen. Any pixel less than the threshold is set to 0 and any pixel greater than the threshold is set to 1.

The steps to segment the image into the various components using canny edge detector are as follows:

The image is filtered with a derivative of Gaussian.

Find the magnitude and orientation of the gradient.

Non-maximum suppression is used to thin wide ridges down to single pixel width.

Perform linking and thresholding to define two thresholds, low and high. Use the high threshold to start edge curves and the low threshold to continue them.

Segment the image into various parts.

In this article, we propose to use fuzzy image segmentation. These components are then used during the classification process that is described later in this module.

Processing the various parts to detect missing parts

We propose to train a standalone object-centric context representation for seeing what is not there. Given an image, our context model can predict where objects should exist, even when no object instances are present. Combined with object detection results, we can find where objects are missing in an image using convolutional NN structure. This technique ignores objects and focuses on context only. For example, here we consider the missing parts in a CPU assembly line.

From computational perspective, an object can be defined as missing in an image region when:

An object detector finds nothing.

A predictor of the object’s typical environment, that is, context, indicates high probability of its existence.

Given an image, we want to detect all such regions efficiently. We need to train this representation to find missing objects. The aim of this module is to scan an image and detect missing parts. In the previous module, as the image is already divided into components, now the step is to develop an automatic algorithm to scan those components to see if any part is missing. Automating such an algorithm especially for components of the assembly line can save a lot of time as well as increase the efficiency and hence increase the revenue of such an industry. The algorithm has to report not only one but also all the missing components.

We propose to create a model that focuses on the context and hence behaves more efficiently. Here as discussed previously, the context is a CPU assembly line. Hence, the model will have to be trained using already available images in this area. The advantage of this is that the model will have to learn just specific images of normal components and hence, this simplifies training.

In order to successfully detect defects, we need to train the system using MI learning. Let F denote the feature space, also known as the instance space. We have the training dataset, T = { ( B 1 , l 1 ) , ( B 2 , l 2 ) , … , ( B n , l n ) } , where { B 1 , … , B n } is a set of bags and each B i is a multiset of n i instances; B i = { B i , 1 , … , B 1 , n i } and L = { l 1 , l 2 , … , l n } , l i ∈ { − 1 , + 1 } are class labels.

Here we assume that a bag represents an object that has to be analyzed for a defect and the label defines whether a feature in the feature space is present or not.

In our case of visual inspection system, all the instances have to be positive for a bag to be labeled positive. The goal of this classification system is to predict the label of an unseen bag

f ( B ) : B → L

A bag represents an image and all the possible defects are the instances. Hence, even if one instance is positive, it means that a defect is present. Our MI learning classifier takes the form

f ( B i ) = { + 1 , if ∃ B i , j ∈ B i | f ( B i , j ) = + 1 − 1 , otherwise

The algorithm for our MI classification is as follows:

Input: A training set, { ( I i , l i ) | 1 ≤ i ≤ n } where I i is an image set.

Output: A trained classifier, f ( B ) .

Step 1. Create an empty training set, T ← { } .

Step 2. For each image I i .

Step 3. Create an empty bag, B i ← { } .

Step 4. Extract all possible defects in E i .

Step 5. For each defect e i , j ∈ E i .

Step 6. Extract instances into B i , j .

Step 7. Add the new instance to the bag, B i = B i ∪ B i , j .

Step 8. Add the new set, T ← T ∪ ( B i , l i ) .

Step 9. f ( B ) is the trained MI learning classifier of the training set T .

The potential defects extracted using segmentation are described in the previous steps. The various components of the image that have been obtained using edge detection algorithms are the potential defects. If any of these potential defects is positive, it signifies that the component is defective, which means that the object has a defect.

Predicting the description of the missing part

Region in-painting is the process of reconstructing the lost or deteriorated parts of images and videos. Here, this process can be used to describe the missing component.

Correcting possible camera occlusion

When we detect an object, one very important factor is that the image should be clear, and that the entire object is properly visible. If these two conditions are not met, we will not be able to conduct any of the previous modules.

Many challenges still exist while detecting an object such as illusion, low visibility, cast shadows, and occlusion of object. Camera occlusion occurs when a camera’s view is obstructed by something. Occlusions occur under two categories: self-occlusion and inter-object occlusion. Hence, the aim of this module is to calibrate the cameras that capture images of the assembled products in such a way that the object is not occluded.

Occlusion handling has been proposed in areas like avoidance of vehicle collisions in urban crowded roads. We propose to use this technique in this article to calibrate the camera to take images of the entire product. If the entire product cannot be included in one image, then more than one image can be used to detect an abnormality in the product.

With the help of occlusion handling, the system will decide if the images produced by a camera can be trusted, or if more than one images (or cameras) are required to produce the desired result. To successfully implement all the modules, various algorithms need to be performed.

Algorithm

Input: A system that already contains trained data, image that has to be processed (Figure 2).

Output: If a product has a defect or not, calibration results of the camera.

Assumption: The camera is automatically calibrated by the system.

Step 1. The image fed into the system is analyzed for occlusion.

Step 2. If the image passes the test for occlusion, then the following steps are performed. Else the camera has to be recalibrated and a new image fed into the system.

Step 3. Perform image segmentation using fuzzy image segmentation.

Step 4. Using the segmented image to divide the image into various components.

Step 5. The various components are now classified using the MI learning classification.

Step 6. If all the components are available, then carry out Step 7. Else carry out Step 9.

Step 7. The positions of the components in the product are checked.

Step 8. If the positions of the components are correct, then carry out Step 11. Else carry out Step 9.

Step 9. The description of the missing part and its position is detected.

Step 10. Output: Defective product.

Step 11. Output: Functioning product.

OPEN IN VIEWER

Figure 2.

Flow chart that shows the various modules.

Experimental results and analysis

In this section, we discuss the experiments performed and the analysis of the proposed system. We compared the results of proposed system with other existing systems. This article is implemented using OpenCV and Python programming language. The experiment has been made to gauge the quality control of a product such as CPU system. Constructing a model that supports quality control, boost productivity, and efficacy by rejecting aberrant products automatically is very essential. Industrial image processing is a broadly used technology which uses distinctive kind of cameras or imaging system that is installed in production streak. In this research, we designed a highly efficient model to automatize CPU system production lines in an industry such that images of the production lines are scanned and any abnormalities in their assembly are pointed out by the model and information about this is transferred to the system administrator via a CPCS network. We have used a machine learning–based approach for classification. Our proposed model not only focuses on just the aberrations but also helps in dynamically configuring the angles from which images of the production taken. The accuracy of our proposed model has been compared with the other existing classifiers. Our proposed methods achieved 92% accuracy while classifying different parts of the production.

Image datasets

In order to evaluate the proposed system, we first created an image dataset consisting of products in the CPU production line. We performed classification using four standard classifiers and compared the results to our proposed system. In our database, we have 50 defective images and 100 non-defective images. The potential defects are extracted as explained previously using fuzzy image segmentation and pattern recognition techniques. The dataset has 150 bags (50 positive and 100 negative). The number of instances is 271 positive instances and 253 negative instances.

In Table 1–3, we compared four state-of-the-art MI learning algorithms^43,44 against the proposed MI learning algorithm. The classifiers that are being evaluated are available in the multiple instance learning (MIL) ⁴⁵ MATLAB toolbox. The simulation has been run several times to predict the accuracy of our model with respect to standard classifiers. First five simulation runs have been displayed in tabular form which is showing the accuracy of the proposed model and other standard classifiers. Table 1 shows the percentage of the correct classification while detecting the defects in screws. Proposed model has the capability to detect the screw either it is defective, missing, or loosely screwed.

OPEN IN VIEWER

Table 1.

The percentage of correct classification when there are defects in screws.

Classifier	Run 1	Run 2	Run 3	Run 4	Run 5
Proposed approach	92.4	93.1	94.2	94.1	93.9
mi-SVM average	90.5	90.4	90.2	91.2	99.6
MIL-Boost	93.2	91.2	90.7	90.9	87.6
MILES	83.5	83.7	79.2	77.6	78.7
MInD Mean-Mean	90.2	89.3	85.7	82.1	83.9

SVM: support vector machine; MIL: multiple instance learning; MILES: multiple-instance learning via embedded instance selection.

OPEN IN VIEWER

Table 2.

The percentage of correct classification when there are defects in labels.

Classifier	Run 1	Run 2	Run 3	Run 4	Run 5
Proposed approach	90.7	91.4	92.3	93.1	92.8
mi-SVM average	78.3	79.7	76.4	77.6	79.2
MIL-Boost	83.2	84.6	81.4	81.9	82.0
MILES	69.4	67.5	70.1	70.8	71.2
MInD Mean-Mean	83.4	83.6	83.7	83.9	84.3

SVM: support vector machine; MIL: multiple instance learning; MILES: multiple-instance learning via embedded instance selection.

OPEN IN VIEWER

Table 3.

The percentage of correct classification when there are defects in sockets.

Classifier	Run 1	Run 2	Run 3	Run 4	Run 5
Proposed approach	92.3	92.9	93.7	94.0	92.8
mi-SVM average	91.9	91.6	90.8	91.7	91.3
MIL-Boost	93.1	92.9	93.4	94.5	93.9
MILES	88.7	88.4	89.5	89.8	87.9
MInD Mean-Mean	93.4	93.6	92.5	93.1	92.9

SVM: support vector machine; MIL: multiple instance learning; MILES: multiple-instance learning via embedded instance selection.

Several simulation runs justify that all classifiers have been examined keenly. Every classifier has different capabilities in detecting different situations. From the results, we can clearly see that the mi-SVM average classifier has the higher accuracy in detecting defective screws or sockets but less capability in detecting defective labels. Similarly, other standard classifiers also have the same kind of abilities in detecting missing, lose, or defective screw and sockets but all systems are not comparatively good at label detection, whereas our proposed model is really efficient in detecting all types of defects such as missing, loose, or defective parts of the production. By observing very keenly the above tables, it can be noted that the test on labels was the most efficient on average. But the various algorithms behave differently for different types of errors. One main reason is the efficiency of the type of segmentation algorithms used for dividing the image into various components.

Figure 3 shows the experimental results for detecting missing and/or defective screw corresponding to five simulation runs to clearly visualize the performance of the proposed system with respect to the various classifiers. We can clearly notice that the proposed system has the highest accuracy compared to other classifiers, whereas mi-SVM average has uncertainty at a fifth run but during the first four runs the accuracy is lower than our proposed system.

OPEN IN VIEWER

Figure 3.

The percentage of correct classification detecting the defective screws.

Figure 4 displays the experimental evaluation to detect the defective labels on the product. The proposed system has the guts to read the labels carefully and to identify the defective piece. Compared to other classifiers, we can clearly see that the proposed system is more capable of reading and identifying the defective label on the product. Similarly, Figure 4 presents the evaluation of defective sockets. The proposed system falters a bit in this area. During the experiment, we observed that the cameras could not capture the sockets on an accurate angle because of which the detection of defective sockets has comparatively less accuracy. Moreover, training data and test data or test system have variation of sockets to check the performance of the system (Figure 5).

OPEN IN VIEWER

Figure 4.

The percentage of correct classification while detecting the defective labels.

OPEN IN VIEWER

Figure 5.

The percentage of correct classification while detecting the defective sockets.

Conclusion

In this article, we have proposed a system that automates the quality assessment of the products produced in an industry. This is done by implementing an industrial image processing–based system in an industry that uses CPCS. The image processing system analyses images of the products taken in the production line for defects. The result of this is then communicated to the concerned authority.

The main challenge in our system is extracting the features of the instances. If appropriate feature extraction mechanisms are not used, it jeopardizes the entire visual inspection system.

For future work, we need to consider the security of the system. Involving the cloud during communication increases the security challenges. Hence, many measures have to be taken to ensure that the entire CPCS system is secure from unwanted third parties.