Development of a real-time Printed Circuit board (PCB) visual inspection system using You Only Look Once (YOLO) and fuzzy logic algorithms

Abstract

Automated visual inspection on PCB boards is a critical process in electronic industries. Misalignment component detection is one of the challenging tasks in the PCB inspection process. Defects during the production process might include missing and misaligned components as well as poor solder connections. Inspection of PCB is therefore required to create practically defect-free products. There are various methods have been developed to perform this task in literature. The significance of this research is to propose an efficient with low-cost system is still require in small scale manufacturing to perform the misalignment or missing component detection on PCB boards. However, an efficient, low-cost system is still required in small-scale manufacturing to perform the misalignment or missing component detection on PCB boards. In this study, a real-time visual inspection system is developed for misalignment component detection. The proposed system consists of hardware and software frameworks. The hardware framework involves the setup of devices and modules. The software framework is composed of pre-processing and post-processing. In pre-processing, image enhancement is applied to remove noises from captured images and You Only Look Once (YOLO) object detector for components detection. Subsequently, the detected components are compared to the corresponding defined pattern using a template-matching algorithm. As experimental shown, the proposed system satisfies the requirement of missing component detection on PCB boards.

Keywords

Surface defect detection visual inspection PCB YOLO fuzzy logic

1 Introduction

As the demands of the global market place ever-increasing pressure on quality, automated visual inspection of industrial items for quality control plays an increasingly important role in the manufacturing process [1, 2]. The majority of the time, visual verification of quality is still done by people [3]. However, there are several issues with a human visual inspection, including inconsistent accuracy, flaws being missed, and a slow inspection rate [4]. In PCB manufacturing, defects during the production process might include missing and misaligned components as well as poor solder connections. Inspection of PCB and PCBA is therefore required to create practically defect-free products [5].

One of the most recent methods for object detection is YOLO [5]. It is a one-shot object identification technique. It is mostly employed in situations when speed is important without losing too much accuracy. When it comes to feature extraction, it makes use of a convolutional neural network [6]. Each picture is first divided into an SS grid of cells for YOLO to function. Each cell is then responsible for determining if an item is present in the image, P(object), and for creating several bounding boxes that are likely to include objects.

Artificial Intelligence (AI) classification methods have been proposed in the classification step, such as Neural Networks (NN) and Fuzzy Logic. Since the performance of the Artificial Intelligence classification techniques considerably depends on their generalization and architecture. This is time-consuming for training and testing phases and may not always find the optimal result for Artificial Neural Networks (ANN). On the other hand, ANN is not a suitable technique for classifying uncertain shapes. Therefore, Fuzzy logic (FL) classification methods mostly apply to classify objects in image processing and pattern recognition [7].

The research problem in this study is dealing with imprecise condition in defect detection and missing printing detection in PCB. On the other hand, Fuzzy logic is based on the observation that people make decisions based on imprecise conditions [10, 11]. Therefore, we choose Fuzzy logic to settle the issue. In this research, a PCB visual inspection system is proposed based on the YOLO object detector and Fuzzy Inference Algorithm. This research is based on image processing techniques in real time. Fuzzy Inference Algorithm as an Artificial Intelligence (AI) classification technique is used for defect-type classification. Then Fuzzy Inference Algorithm, including Mamdani and Sugeno models, is implemented to classify the footprints, and the performance comparison is evaluated. As a result, the major contributions of this research are: (i) An automated visual inspection system on the PCB using an object detector, (ii) a defect classification on PCB using YOLO object detection and Fuzzy Inference Algorithm. This study is based on low-cost hardware setup and effective approaches for PCB inspection. For this reason, it can be considered unique in this research scope.

2 Literature review

This section discusses previous studies on surface inspection. In [8], the identification and categorization of different flaws in steel surfaces are made using the YOLO network. The network can also extract the coordinates of the flaws, which provides information on the position and size of each defect found. They trained on a dataset with six different types of flaws on steel surfaces using the YOLOV3 detector.

Wu et al. [9], an automatic visual inspection technique was presented using PCB attributes analysis, including the location, form, and logical aspects of the solder joints on the PCB [3]. The proposed method can be used to spot a variety of PCB flaws, including excess solder, incorrect or missing components, broken components, and no solder at all. The characteristics will be removed based on a number of places, and the solder connects’ design after the solder connections’ placement has been established. To determine the occupancy ratio of the area, color, center of gravity, and continuous pixels, they evaluate solder joints. The logical features are extracted by analysis of the close connections between shape features, position features, and color-dispersing features.

Matsushima et al. [10] presented the PCB solder joints examination using a neural network. Principal component analysis (PCA) is used to extract the features from the input data for the learning phase and inspection phases. The camera angle and the angle of the light source influence the shooting conditions. The learning and inspection phases make up the neural network visual inspection system. Two groups were created from the 120 excellent photographs and the 120 poor ones. The first dataset is for learning, while the second dataset is for inspection. In the learning phase, the neural network system produces two outputs, one representing the sample’s defect degree and the other its good degree based on the inputs it gets from a good or defective sample.

In [15], by capturing the reference PCB picture and a test PCB image, this method is used to discover those flaws. The RGB loaded image on MATLAB then displays the missing or out-of-place components in the foreground plane (FG) and the flaws in the PCB surface. The background modeling ideas in addiction are also explained utilizing the Mixture of Gaussian (MoG) classical technique. This paper also includes results relating to compression image average and detecting efficacy.

Fridman et al, [16] introduced offer ChangeChip, a computer vision (CV) and UL-based automated and integrated change detection solution for PCB faults, from soldering flaws to missing or misaligned electrical components. They use an unsupervised change detection between photos of a golden PCB (reference) and the examined PCB under various conditions to accomplish high-quality flaw identification. In addition, CD-PCB is presented in this study, a synthetic labeled dataset of 20 pairs of PCB pictures for testing defect identification algorithms.

In this study, compared to other works, we study utilized low-cost devices for Hardware setup. Furthermore, we use efficient deep learning model for YOLO based on CPU processing. Therefore, the use of low-cost and low computation cost method, the method presents effective and low-cost inspection method.

3 Proposed system

In this study, an automated visual inspection system is proposed, which is consisted of hardware and software frameworks. The hardware framework involves devices and components to set up hardware for the proposed automated visual inspection system. Subsequently, the software framework presents modules and methods for the system.

3.1 Hardware framework

The hardware framework consists of the structure of devices and modules set up for the proposed visual inspection system. Figure 1 shows the hardware framework. As shown in the Fig. 1, we utilized low-cost devices for hardware setup.

Fig. 1

Hardware framework in the proposed visual inspection system.

3.2 Software framework

The software framework involves pre-processing and post-processing stages. Figure 2 shows the steps in a proposed software framework.

Fig. 2

Steps in proposed software framework.

3.2.1 Preprocessing

This stage involves image enhancement and a Yolo object detector. The details of each are discussed as follows.

i. Image enhancement

Image enhancement is a single step in processing digital images that modifies and prepares the pixel values to create a more acceptable form for other processes. The removal of undesired reflections from the system and disturbances, which increase data size and processing time, requires the employment of image-enhancing techniques.

ii. YOLO object detector

The YOLO object detector first creates a dataset to train a model. The dataset is created by gathering images from different resources, such as the internet, and captured images by our imaging devices. In the training stage, several images were labeled according to the object name. As for the training model similar to Hatab et al. [8], a YOLOV3 is trained on our dataset and tested on PCB components. Figure 3 shows some sample images from PCB components for dataset preparation. These images were acquired from the internet and our captured image process. The samples images are components from PCB, including the footprints such as Variable Resistor, Capacitor, LED, IC, and Transistor.

Fig. 3

The sample images for dataset preparation.

3.2.2 Post-processing

This stage discusses on defect classification process on PCB components. To categorize a stage for this classification, a collection of known traits may be computed as soon as they are extracted. In this study, the area and perimeter as geometrical features are chosen for the classifying footprint. Fuzzy logic-based algorithms, such as the Mamdani and Sugeno models, are employed in this stage to divide the footprints into four groups for each type of footprint (25, 50, 75, and 100 percent).

i. Mamdani Model

Using fuzzy systems provides advantages in several industrial vision applications [7]. This study uses two geometric features— Area and Perimeter— as the basis for our fuzzy analysis. The Mamdani and Sugeno fuzzy, as described above, only has one output, O, and two inputs, A (Area) and P (Perimeter) (output). Here, we demonstrate the linguistic value of the fuzzy set for a certain sort of component (a capacitor) on a printed circuit board. Table 1 shows the Capacitor’s fuzzy set variable.

Table 1
The variable of fuzzy set of Capacitor

Linguistic Variable Linguistic Value Description Numerical Range

Perimeter, P P1 Between 0-less25 Cap 0 – 165

P2 Between 25-less50 Cap 155 - 355

P3 Between 50-less75 Cap 295 - 500

P4 Between75,less100Cap 490 - 700

Area, A A1 Less100 Cap 0 - 1650

A2 100 Cap 1590 - 2100

Output,O O1 25% Cap 0 – 0.25

O2 50% Cap 0.23 – 0.5

O3 75% Cap 0.47 – 0.75

O4 100% Cap 0.73 - 1

Linguistic Variable	Linguistic Value	Description	Numerical Range
Perimeter, P	P1	Between 0-less25 Cap	0 – 165
	P2	Between 25-less50 Cap	155 - 355
	P3	Between 50-less75 Cap	295 - 500
	P4	Between75,less100Cap	490 - 700
Area, A	A1	Less100 Cap	0 - 1650
	A2	100 Cap	1590 - 2100
Output,O	O1	25% Cap	0 – 0.25
	O2	50% Cap	0.23 – 0.5
	O3	75% Cap	0.47 – 0.75
	O4	100% Cap	0.73 - 1

ii. Sugeno Model

The Sugeno model resembles the Mamdani approach in many ways. Fuzzifying the inputs and using the fuzzy operator are the first two steps in the fuzzy inference process, and they are identical. Sugeno and Mamdani vary primarily in that Sugeno’s output membership functions are either linear or constant.

4 Results and discussion

This section presents the experimental results for pre-processing and post-processing stages.

4.1 Pre-processing results

Image enhancement and YOLO object detector results are shown in this section. Table 2 shows the results of this stage for object detection for a particular component with different percent of the missed area.

Table 2
The results of template matching on the capacitor class

Percentage of the components Detected object

Miss

25%

50%

75%

100%

Percentage of the components	Detected object
Miss
25%
50%
75%
100%

The performance of the proposed method is evaluated for Mamdani and Sugeno algorithms. Furthermore, since the method is implemented for some missing printing process, there is not much more studies to consider these challenges (as mentioned in above cell). Therefore, two Fuzzy based algorithms are experimented and the results are presented. Tables 3 and 4 show how the results and performance percentage. In printing process of PCB products, sometimes the location of electronic components is not printed properly. The printing process in some cases printed proportional with relative values such as 25%, 50%, and etc. Therefore, inspired from real scenario, er tested our method to cover these challenges. Tables 3–6 are presented the result of our proposed method to measure the performance in these scenarios.

Table 3

The number of correct classifications in Mamdani method

Type of footprint	25% of footprint	50% of footprint	75% of footprint	100% of footprint
Capacitor	23	69	46	23
Variable Resistor	23	91	17	242
IC	23	89	23	23
LED	23	46	45	203
Transistor	20	39	21	827

Table 4

The performance of Mamdani method

Type of footprint	25% of footprint	50% of footprint	75% of footprint	100% of footprint
Capacitor	100%	100%	100%	100%
Variable Resistor	100%	99%	67%	96%
IC	100%	96%	100%	100%
LED	100%	100%	98%	98%
Transistor	87%	85%	91%	97%

Table 5

The number of correct classifications in Sugeno

Type of footprint	25% of footprint	50% of footprint	75% of footprint	100% of footprint
Capacitor	18	68	42	23
Variable Resistor	21	91	19	253
IC	23	88	22	23
LED	23	46	46	197
Transistor	22	44	20	828

Table 6

The performance of Sugeno method

Type of footprint	25% of footprint	50% of footprint	75% of footprint	100% of footprint
Capacitor	78%	99%	91%	100%
Variable Resistor	91%	99%	83%	100%
IC	100%	96%	96%	100%
LED	100%	100%	100%	93%
Transistor	96%	96%	87%	97%

4.2 Mamdani model evaluation

The Mamdani method’s effectiveness on PCBs is assessed using the 207 images as our dataset test. In this case, a real-time inspection is implemented as a classifier in the real world using the Mamdani approach. When the suggested method properly categorizes the form in a segmented image, the accuracy number equals the percentage of footprints. Each component’s footprint is assessed independently in accordance with the suggested segmentation. At the conclusion, we demonstrate the effectiveness of the Mamdani approach on each type of footprint using the equation below. $\begin{matrix} Accuracy of Mamdani \\ = \frac{\sum correct classification of footprints}{number of related footprints} \times 100 \end{matrix}$

In the dataset test, we used the Mamdani approach to compute the number of accurate classifications. Tables 3 and 4 show how the results and performance percentage.

4.3 Sugeno model evaluation

Similar to the Mamdani model evaluation, the dataset test images are used to evaluate the Sugeno model’s performance on PCBs. Tables 5 and 6 present the number of correct classifications and the corresponding percentage pf performances for the Sugeno model.

Always select A4 as the paper size in the document settings. The type area is 16.2 cm wide and 21.5 cm long. This area must be used to the maximum, and at the same time must not be exceeded. The following margin settings for A4 size paper will produce the correct result: top 4.5 cm; bottom: 3.7 cm; left and right: 2.4 cm.

The text should be in a double column format with a space of 10 mm between the two 76 mm wide columns.

As shown in Table 3 the number of correct classification or true positive are calculated for Mamdani fuzzy inference method. These calculation for each footprint category of 25, 50, 75 and 100 percentages. Table 4 presents the performance percentage for the calculated values of Table 3 for Mamdani method. Moreover, Table 5, 6 shows the number of correct classification or true positive calculation and corresponding performance percentage for Sugeno fuzzy inference method. In average, as experimental results shown the Mamdani method presents better results compared to Sugeno method because of Mamdani fuzzy inference provides more interpretability and transparency by generating linguistic rules that can be easily understood by humans, while Sugeno generates crisp outputs that can be difficult to interpret.

4.4 Performance comparison

This section presents performance comparison for our proposed method and other related works. To fair comparison between our proposed method and other methods, we use another performance metric as Mean Average Precision (mAP) which is based on precision and recall measurements. Following equation is used to measure the mAP, $mAP = \frac{\sum_{i = 1}^{C} {AP}_{i}}{C}$ where the average of the Average Precision (AP) values of C different defects, and it reflects the accuracy of defect detection.

We experiment our proposed method using the dataset in [17] and collect the results. Inspired from the obtained experimental and collected data reported in [17], this comparison is conducted. Table 7 shows the obtained results for the proposed methods and other methods.

Table 7
Comparison of detection accuracy results between the proposed and other methods

Method Accuracy rate

Faster RCNN 91%

RetinaNet 90%

Yolov3 71%

Faster-RCNN and Feature Pyramid Network 95%

The proposed method 96%

Method	Accuracy rate
Faster RCNN	91%
RetinaNet	90%
Yolov3	71%
Faster-RCNN and Feature Pyramid Network	95%
The proposed method	96%

As experimental results and performance comparison shown in Table 7, the proposed presents better results compared to other existing methods.

5 Conclusion

In this work, a real-time visual inspection system is proposed to identify misaligned components on PCB. Hardware and software frameworks structured for the proposed system. Device and module configuration are required for the hardware framework. Pre-processing and post-processing make up the software framework. Image enhancement is used in the pre-processing stage to remove noise from the collected images, and the YOLO object detector is used to identify components. The identified components are then evaluated against the associated prescribed pattern using classification strategies based on fuzzy logic models. This study utilized low-cost devices for Hardware setup. Furthermore, we use efficient deep learning model for YOLO based on CPU processing. Therefore, the use of low-cost and low computation cost method, the method presents effective and low-cost inspection method. As experimental results and performance evaluation are shown the proposed system meets the need of missing component detection on PCB boards. The limitation of this study is missing detection of components and footprints in more complicated PCBs. For future study, the current method cane be improved for more complex PCBs. Furthermore, other lightweight deep leaning based frameworks can be investigated for PCB defect detection.

References

Liu

and Qu

, Machine vision based online detection of PCB defect, Microprocessors and Microsystems 82 (2021), 103807.

Liu

et al., Machine vision based condition monitoring and fault diagnosis of machine tools using information from machined surface texture: A review, Mechanical Systems and Signal Processing 164 (2022), 108068.

Kocako\c

İ.D.

, c and G.B. Türkölmez, Selection of solder paste inspection machines by multi-criteria decision analysis, Journal of Management and Economic Studies 4(1) (2022), 47–58.

et al. A Real-Time Vision System for Defect Detection in Printed Matter and Its Key Technologies. 2007. IEEE.

Anitha

and Rao

, A survey on defect detection in bare PCB and assembled PCB using image processing techniques. in International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET). 2017. IEEE.

Redmon

and Farhadi

, YOLOv3: An Incremental Improvement [J]. Arxiv. org, 2018.

Kou

et al., Development of a YOLO-V3-based model for detecting defects on steel strip surface, Measurement 182 (2021), 109454.

Hashim

H.S.

, Abdullah

S.N.H.S.

and Prabuwono

A.S.

, Automated visual inspection for metal parts based on morphology and fuzzy rules. in Computer Applications and Industrial Electronics (ICCAIE), 2010 International Conference on. 2010.

Chhabra

, Sharan

and Kumar

, A fuzzy cognitive map of the quality of user experience determinants in mobile application design, Journal of Intelligent & Fuzzy Systems, (Preprint): p. 1–23.

10.

Chithaluru

et al., An enhanced energy-efficient fuzzy-based cognitive radio scheme for IoT, Neural Computing and Applications 34(21) (2022), 19193–19215.

11.

Raheja

et al., A smart intuitionistic fuzzy-based framework for round-robin short-term scheduler, The Journal of Supercomputing (2022), 1–25.

12.

Hatab

, Malekmohamadi

and Amira

, Surface defect detection using YOLO network. In Proceedings of SAI Intelligent Systems Conference. 2020. Springer.

13.

et al. Automated visual inspection of surface mounted chip components. in International Conference on Mechatronics and Automation. 2010. China: IEEE.

14.

Matsushima

et al., Visual inspection of soldering joints by neural network with multi-angle view and principal component analysis, Service Robotics and Mechatronics (2010), 329–334.

15.

Bonello

D.K.

, Iano

and Neto

U.B.

, A Novel Approach to the PCBAs Defects Detection Using Background Algorithm. 2020.

16.

Fridman

, Rusanovsky

and Oren

, ChangeChip: A reference-based unsupervised change detection for PCB defect detection. In IEEE Physical Assurance and Inspection of Electronics (PAINE). 2021. IEEE.

17.

and Wang

, Detection of PCB surface defects with improved faster-RCNN and feature pyramid network, IEEE Access 8 (2020), 108335–108345.