BOSF-SVM: A thermal image-based fault diagnosis method of circuit boards

Abstract

In recent years, the lack of thermal images and the difficulty of thermal feature extraction have led to low accuracy and efficiency in the fault diagnosis of circuit boards using thermal images. To address the problem, this paper presents a simple and efficient intelligent fault diagnosis method combined with computer vision, namely the bag-of-SURF-features support vector machine (BOSF-SVM). Firstly, an improved BOF feature extraction based on SURF is proposed. The preliminary fault features of the abnormally hot components are extracted by the speeded-up robust features algorithm (SURF). In order to extract the ultimate fault features, the preliminary fault features are clustered into K clusters by K-means and substituted into the bag-of-features model (BOF) to generate a bag-of-SURF-feature vector (BOSF) for each image. Then, all of the BOSF vectors are fed into SVM to train the fault classification model. Finally, extensive experiments are conducted on two homemade thermal image datasets of circuit board faults. Experimental results show that the proposed method is effective in extracting the thermal fault features of components and reducing misdiagnosis and underdiagnosis. Also, it is economical and fast, facilitating savings in labour costs and computing resources in industrial production.

Keywords

Thermal images circuit boards fault diagnosis bag-of-features support vector machine

1 Introduction

Circuit abnormalities caused by component failures can make the entire system fail or even cause catastrophic accidents [1, 2]. Rapid and accurate identification of circuit board faults is beneficial to improving maintenance efficiency and restoring equipment operation as early as possible. However, it is difficult to obtain fault data in practical engineering applications due to the relative reliability of circuit boards and the complexity and variability of the operating environment, which poses a serious challenge to the fault diagnosis of circuit boards [3].

In recent years, infrared thermography has become an emerging non-destructive testing method, which is of great interest to researchers in the field of fault diagnosis due to its advantages of no contact, no downtime, simple operation, and temperature visualization [4 –6]. The earliest methods of circuit board fault diagnosis using infrared thermography were the “difference method” and the “sequence method”. These methods only require comparison with standard boards and focus on image processing, including alignment, fusion, and segmentation of visible and thermal images, with the aim of visualizing the location of components [7 –9]. The “difference method”, in which the difference at a given moment between the thermal images of the standard board and the board under test is compared with a set temperature threshold to identify obvious thermal faults; the “sequence method” is used for fault diagnosis by comparing the temperature profiles of the components in the thermal sequence images of the standard board and the board under test [10]. Nevertheless, the “difference method” is prone to serious misdiagnosis and underdiagnosis as it usually relies on subjective human experience when setting thresholds, while the “sequence method” is complex to operate and data-intensive due to collecting temperature profiles of all components.

With the development of artificial intelligence, machine learning has become the dominant method in the field of infrared thermography-based circuit board fault diagnosis. The diagnostic process is divided into image preprocessing, temperature feature extraction, and pattern recognition. Wang et al. and Hao et al. did a lot of research on infrared thermography-based fault diagnosis of airborne circuit boards, using the preprocessed thermal sequence images to obtain the temperature feature matrix of the components, extracting multiple temperature features reflecting the faults and a multi-level intelligent fault diagnosis model is finally established by combining SVM classifier and D-S evidence theory, which effectively improves the efficiency and accuracy of identifying different fault states of circuit boards [11, 12]. Mehra R used two-dimensional discrete wavelet transform (2D-DWT) and principal component analysis (PCA) for feature extraction and dimensionality reduction of the thermal images of the circuit boards respectively and then fed the feature vectors into SVM to finally diagnose the “health” and “fault” states of circuit boards [13]. Although machine learning-based diagnostic methods have achieved good results, it currently suffers from the following problems: firstly, the fault diagnosis process is rather cumbersome, involving techniques such as alignment, fusion, and segmentation of thermal and visible images, which is relatively inefficient. Secondly, as the temperature sensitivity of a thermal imaging camera affects the accuracy of fault diagnosis, an accurate temperature matrix needs to be obtained before extracting temperature features, which greatly increases the hardware cost of purchasing infrared equipment.

Luckily, with the continuous optimization of image feature extraction algorithms, computer vision-based classification methods are widely used in many fields and have obtained better classification results. Ahmed W et al. utilized the SURF algorithm to extract features and the K-means clustering algorithm to build a visual feature bag and then combined it with a multi-class shallow classifier to classify the three health states based on the infrared images of the PV panel with an accuracy of 97% [14]. K Bakour et al. proposed a new generic image classification method based on local and global image features, a bag of visual words, and machine learning techniques for Android malware classification [15]. Li et al. proposed a noisy partial discharge (PD) pattern recognition method based on the SURF algorithm and improved SVM, which effectively improved the accuracy of PD recognition [16]. Arora G et al. proposed a BOF and SVM-based method for the early diagnosis of skin cancer, resulting in a 3% improvement in diagnostic performance [17]. Therefore, aiming at the shortcomings of current machine learning-based circuit board fault diagnosis methods, the paper introduces a computer vision approach into the machine learning-based diagnostic method. It integrates the bag-of-features model (BOF) with the speeded-up robust features algorithm (SURF) to extract the BOSF fault features from thermal images of circuit boards and trains a lightweight multi-classification SVM model to diagnose faults.

The goal of this study is to design an optimal model for circuit board fault classification that satisfies fast diagnosis while maximizing all evaluation indicators, especially accuracy. The objectives can be formulated as follows: $Evaluation indicator : \max_{f} M (f) > 99 %$ (1) $Test time : min_{f} T (f) < 0.03 s$ (2) $s . t . Images : 300 ⩽ f ⩽ 600$ (3) where M (f) is the evaluation indicators, as illustrated in Eqs. (14)–(17), including precision, recall, f1-score, and accuracy, T (f) represents the test time per thermal image, and f is the number of thermal images. Therefore, under the assumption that there is a significant change in the temperature of the component in its fault state compared to its normal state and no sufficient thermal images, the performance of the model will be comprehensively judged by Eqs. (1)–(3) with accuracy above 99% and test time less than 0.03 s.

The main contributions of this paper are summarized as follows.

A computer vision approach is introduced to simplify the diagnostic process. The BOSF feature extraction provides a more accurate and robust representation of thermal faults, thus avoiding complex image alignment, image segmentation, and component temperature extraction operations.

The multi-classification SVM model with strong generalization used is well suited to the situation of small samples and high-dimensional data in circuit board thermal images.

The superiority of this method is that by replacing manual diagnosis with an intelligent method, it greatly saves computational resources in industrial production, improves the accuracy and efficiency of fault diagnosis, and has strong robustness and real-time performance.

2 Related theory

2.1 SURF algorithm

SURF, originally proposed by Bay et al. [19] in 2006, is a robust and fast algorithm for local feature points detection and description and is the speedy version of Scale-invariant feature transform (SIFT). It is widely used in object recognition, image retrieval and image registration due to its advantages of fast computational speed and excellent scale, rotation and noise invariance [20 –22]. SURF consists of two main phases: feature point detection and feature point descriptor generation.

The first phase uses the Hessian matrix, when its discriminant takes a local maximum to determine the feature points; The second phase uses the Harr wavelet to determine the principal direction of the feature points and generate 64-dimensional descriptors.

2.2 BOF model

BOF is derived from the Bag-of-Words model in the field of text retrieval and was transplanted to image processing by CSURKA G et al. [23] in 2004. The BOF model is composed of three steps.

Detect interesting points in the image and construct descriptors for each point using a local feature algorithm.

Use a clustering algorithm, such as K-Means, to cluster the descriptor data set into multiple classes.

Construct a frequency histogram for each image based on the frequency of descriptors in the image that belong to each clustering class.

2.3 SVM classifier

SVM is a machine learning algorithm based on statistical learning theory and structural risk minimization, mainly used to solve binary classification problems. The core idea is to find a marginal maximization decision boundary to separate two classes of samples [24].

It is also widely used in the field of multi-classification because of its powerful performance like neural networks. It will be converted to several binary classification problems by one-versus-one strategy (ovo) or one-versus-rest strategy (ovr) when solving multi-classification problems [25,26, 25,26].

3 A thermal images-based fault diagnosis method: BOSF-SVM

When a circuit board is energized, the energy-consuming components heat up and produce thermal radiation which is detected by the thermal imaging camera and converted into thermal images to show its temperature field. The brightness and darkness of the thermal image correspond to the intensity of the thermal radiation, which is different in normal and faulty conditions [18]. When a component is short-circuited, the current through it increases, resulting in an increase in thermal radiation, a brighter thermal image, expansion of the heated area, and more visible texture; when a component suffers from open-circuited or a dummy connection, the current through it decreases, resulting in less thermal radiation, a darker thermal image and a less distinct texture.

Consequently, due to the above characteristics of circuit board thermal images, this paper correlates the texture features of thermal images with different fault states of circuit boards and proposes the fusion of BOF and SURF as a local feature descriptor to extract texture features of the faulty components from thermal images according to Section 3.3, and then applies a multi-classification SVM model as a classifier to realize circuit board fault diagnosis as noted in section 3.4. In the end, a circuit board thermal image fault diagnosis model with good generalization capability is constructed. Figure 1 gives the flowchart for the method proposed in this paper. The specific fault diagnosis algorithm in this paper is given in Table 1.

Fig. 1

Flowchart of the proposed method.

Table 1

BOSF-SVM modelling algorithm

Algorithm 1. BOSF-SVM modelling algorithm.
Input:
IR(Thermal images of circuit boards)
Output:
Fault classification results.
Require:
k (The number of clusters found by k-means)
C,γ(The parameter search scope of SVM for grid searching)
Procedure:
I: Image preprocessing.
cv2 . resize (IR) ←Downsample the IR.
cv2 . cvtcolor (IR_resized) ←Grey out the resized IR.
cv2 . createCLAHE (IR_gray) ←Enhance contrast while suppressing noise.
II: Extract the preliminary fault features.
For x_i ∈ IR do
p_i ←Find the SURF feature points set.
m_i ←Compute the SURF feature matrix.
End
m_train, m_test ← Construct SURF feature matrix of IR_train, IR_test.
III: Extract the ultimate fault features.
k centers←K-means (m_train, k) to obtain a visual vocabulary.
For m_ij ∈ m_i do
label← $z_{ij} = arg min_{t} ∥ m_{ij} - c_{t} ∥^{2}$ f_k
End
BOSF_i ← get the occurrence frequency of each visual word.
BOSF_train, BOSF_test ←Construct BOSF fault feature matrices.
IV: Fault diagnosis.
the optimal model← GridSearchCV (SVM (BOSF_train, C, γ)) with 5-fold cross-validation
SVM (BOSF_test) ←Evaluate the optimal model
Return Fault classification results

3.1 Thermal image datasets acquisition

Given that there is no publicly available circuit board thermal image datasets, this paper uses two homemade datasets for the fault identification experiments. Experiments were conducted on the human reaction speed test board and the simulated remote control fan board, abbreviated as fan board and speed board, as shown in Fig. 1, introducing faults to the normal board by making artificial faults. The speed board includes 5 artificially set fault states and 1 normal state; the fan board includes 2 artificially set fault states and 1 normal state. The experimental setup has been shown in Fig. 1 with a thermal imaging camera, circuit boards, a programmable power supply, a PC, storage hard disk and communication buses. Using the InfiRay Xtherm II T2L thermal imaging camera to capture thermal images of both boards at full capacity after a period of power-up, 100 saturated thermal images were taken for each state of both boards at the same ambient temperature, 600 for the speed board and 300 for the fan board.

3.2 Image preprocessing

The image size was turned to 1420*1080 since the thermal images were taken with the thermal imaging camera connected to the mobile phone and transferred to the PC when stored later. To reduce the computational complexity, the thermal images were first downscaled to 355*270 and then greyed out. In the end, contrast-limited adaptive histogram equalization was adopted to enhance contrast, thus highlighting the features of the region of interest in thermal images while playing a role in noise suppression.

3.3 BOSF feature extraction

BOSF is constructed by the fusion of the BOF model and the SURF algorithm. Firstly, it uses SURF to extract the preliminary fault features of the circuit board thermal images. It represents an image as a collection of local SURF feature descriptors. Then, a visual vocabulary is generated using K-means and the descriptors are assigned to the visual words according to the nearest neighbour principle for thermal image vectorization and dimensional reduction.

Given thermal images IR = {x_i : i = 1, 2, 3, . . . , N}, for each thermal image x_i, the calculation of its BOSF vector BOSF_i can be expressed by 3 sub-calculations:

1) It selects the points of interest of a circuit board thermal image using the Hessian matrix: $H (x, σ) = [\begin{matrix} L_{xx} (x, σ) & L_{xy} (x, σ) \\ L_{xy} (x, σ) & L_{yy} (x, σ) \end{matrix}]$ (4) where L_xx (x, σ) is the convolution of the Gaussian second order derivative with the image I in point x = (x, y) at scale σ, and similarly for L_yy (x, σ) and L_yy (x, σ). To improve speed, SURF utilized a box filter to approximate the Gaussian filter. Also, it introduces a weighting factor of 0.9 to reduce the deviation caused by the approximation. Thus, the determinant value of the Hessian matrix can be approximated by Eq. (5). $det (H_{approx}) = D_{xx} D_{yy} - {(0.9 D_{xy})}^{2}$ (5)

When the discriminant of the Hessian matrix achieves a local maximum, the current point is determined to be a key point which is brighter or darker point than other points in the surrounding neighbourhood. Thus, p_i, a collection of SURF feature points, is obtained.

2) To make the feature point rotationally invariant, SURF determines the principal direction vector by calculating the Haar wavelet response values on the x and y axes for all points in the neighbourhood of the feature point. After obtaining the principal directions of the feature points, a square area with a side length of 20σ is determined with the feature point as the center and the principal direction as the x-axis, the square area is divided into 4*4 sub-regions, and the Haar wavelet response values d_x and d_y in the horizontal and vertical directions are calculated for each area. Then, we represent the sub-region by a four-dimensional vector as Eq. (6). $V = (\sum d_{x}, \sum d_{y}, \sum | d_{x} |, \sum | d_{y} |)$ (6)

Finally, the vectors of the 16 sub-regions are concatenated to form a 64-dimensional descriptor. A preliminary feature matrix m_i is formed by all the descriptors from a thermal image.

3) K-means is used to cluster all SURF descriptors of the training set m_train 10 times, each time randomly selecting the initial clustering centers and taking the best clustering result as the final clustering result, thus obtaining k clustering centers with which to construct a fault visual vocabulary of size k as ${c_{1}, c_{2}, . . ., c_{k}}, c_{t} \in ℝ^{n}$ .Then, calculate the Euclidean distance between the SURF descriptors for each thermal image in the datasets and the k visual words, and assign each descriptor to the nearest visual word according to: $z_{ij} = arg min_{t} ∥ m_{ij} - c_{t} ∥^{2}$ (7) where m_ij is the jth SURF descriptor of the preliminary feature matrix m_i, c_t (t = 1, 2, . . . , k) is the tth visual word, and z_ij ∈ 1, 2, . . . , k is the label of m_ij. Finally, each image is represented as a BOSF vector by counting the frequency of occurrence of the k visual words in each image, as shown in Eq. (8). ${BOSF}_{i} = [f_{1}, f_{2}, . . ., f_{k}]$ (8) where f_k indicates the number of occurrences of the kth visual word.

3.4 Fault diagnosis

The lightweight and fast multi-classification SVM model is adopted to meet the requirement of real-time processing. The BOSF vectors obtained after feature extraction was used as the final valid input to the multi-classification SVM model. The “ovr” strategy is employed to build the same number of SVM classifiers as the fault types, where the goal of each SVM is to maximize the margin of decision boundary to separate one class from the rest samples. Each binary classifier is calculated as follows.

Given N training samples where ${x_{i}} \in ℝ^{d}$ is the input vector and y_i ∈ {+1, - 1} indicates the corresponding desired output vector. The optimal classification hyperplane is computed by Eq. (9). $w \cdot Φ (x_{i}) + b = 0$ (9) where w and b are the hyperplane parameters, representing the weight vector and offset respectively. The loss function is defined as Eq. (10). $\begin{matrix} Q_{min} (w) = \frac{| | w | |^{2}}{2} + C \sum_{i = 1}^{N} ξ_{i} \\ s . t . y_{i} (w Φ (x_{i}) + b) ⩾ 1 - ξ_{i}, ξ_{i} ⩾ 0 \end{matrix}$ (10) where ξ_i is the slack variable, which is introduced to allow samples to be misclassified; C is the penalty coefficient, which indicates the degree of penalty for misclassified samples and is one of the parameters that affect the classification performance of SVM.

Introduce the Lagrange multiplier method to solve the optimization problem. When the Lagrangian function satisfies the KKT condition, the problem is converted into solving the maximum of the Lagrangian pairwise functions as shown in Eq. (11). $\begin{matrix} L_{max} (α) = \sum_{i = 1}^{N} α_{i} - \frac{1}{2} \sum_{i, j = 1}^{N} α_{i} α_{j} y_{j} y_{j} (Φ (x_{i}) \cdot Φ (x_{j})) \\ s . t . 0 ⩽ α_{i} ⩽ C, \sum_{i = 1}^{N} α_{i} y_{i} = 0 \end{matrix}$ (11) where 32 × 32 is the Lagrange multiplier. By solving the above optimization problem, the final SVM classifier is as Eq.(12): $f x = sgn (\sum_{i = 1}^{N} α_{i} y_{i} (Φ (x) \cdot Φ (x_{i})) + b)$ (12) where x is any test sample, sgn (h) is a symbolic function, h > 0 returns 1 and h < 0 returns -1. To avoid the “curse of dimensionality”, a kernel trick is introduced. This paper uses the RBF kernel function as shown in Eq. (13) since it performs well on most datasets. $K (x, x_{i}) = exp (- γ | | x - x_{i} | |^{2})$ (13) where γ is a Gaussian kernel parameter that affects the fineness of the sample classification. Therefore, it is necessary to choose the appropriate C and γ to obtain the superior performance of SVM.

4 Experiment and evaluation

The experiments are based on a Python 3.7 environment, using AMD Ryzen 5 5600H, 16GB RAM, NVIDIA GeForce RTX 3050Ti, mainly using libraries such as scikit-learn and OpenCV.

4.1 Introduction of data sources

To verify the effectiveness and superiority of the proposed method, this paper uses two homemade thermal image datasets for the circuit board fault identification experiments. The dataset will be parted into training and testing (30 : 70) as shown in Tables 2 3.

Table 2
Fan dataset division

State code Thermal images Training set Test set Fault classes

0 30 70 ICbreak

1 30 70 ICshort

2 30 70 ICnormal

State code	Training set	Test set	Fault classes
0	30	70	ICbreak
1	30	70	ICshort
2	30	70	ICnormal

Table 3

Speed dataset division

State code	Thermal images	Training set	Test set	Fault classes
0		30	70	IC1 short
1		30	70	IC2 short
2		30	70	IC3 short
3		30	70	IC4 short
4		30	70	normal
5		30	70	R2R11break

4.2 Evaluation indicators

Considering that the data acquired in real-world industrial production may have the issue of uneven samples of different classes, thus for a multi-classification problem with m classes of samples, this paper uses a weighted-average calculation to evaluate the classification performance of the fault diagnosis model on both datasets. The range for all four evaluation indicators is [0,1]. A higher value means better performance. $precision = \sum_{i = 1}^{m} \frac{{TP}_{i}}{{TP}_{i} + {FP}_{i}} \times p_{i}$ (14) $recall = \sum_{i = 1}^{m} \frac{{TP}_{i}}{{TP}_{i} + {FN}_{i}} \times p_{i}$ (15) $f 1 - score = \sum_{i = 1}^{m} \frac{2 {TP}_{i}}{2 {TP}_{i} + {FN}_{i} + {FP}_{i}} \times p_{i}$ (16) $accuracy = \sum_{i = 1}^{m} \frac{{TP}_{i} + {FN}_{i}}{{TP}_{i} + {TN}_{i} + {FP}_{i} + {FN}_{i}} \times p_{i}$ (17) where $p_{i} = \frac{{TP}_{i} + {FN}_{i}}{\sum_{i = 1}^{m} ({TP}_{i} + {FN}_{i})}$ is the weight of class i, indicating the proportion of samples from class i to the total samples. TP_i, TN_i, FP_i and FN_i are the true positive, true negative, false positive and false negative cases defined for class i with itself as the positive class and all other classes as the negative class, respectively. Eqs. (14)–(17) reflects the misdiagnosis, underdiagnosis, the balance of them, and accuracy on the circuit board faults, respectively.

4.3 Impact of vocabulary size on diagnosis performance

The BOSF-SVM method extracts the fault features for each thermal image through the BOSF model to obtain a BOSF vector of length k, which is used as the actual input to the final multi-classification SVM model to achieve fault classification. An important parameter of the BOSF-SVM is the vocabulary size K, whose value will directly affect the generalization of the classifier. Hence, in this paper, different k values were selected in steps of 50 within the range of [50,300] and the effect of k on the average classification accuracy was observed on the test set.

As can be seen from Fig. 2, the speed dataset and fan dataset exhibit the highest fault diagnosis accuracy at k = 200 and k = 100, with 99.76% and 99.05% respectively, which is due to the limited valid feature points in the faulty component region of the thermal images. As a result, the accuracy of the algorithm no longer increases when k exceeds a certain value and even decreases due to classification errors caused by redundant descriptions.

Fig. 2

The average accuracy of fault classification for different k values (a) The speed dataset (b)The fan dataset.

4.4 Comparison of different feature extraction methods

To verify the superiority of the feature extraction method in this paper, it was compared with four image feature extraction methods, namely gray-level histogram statistical features (GHSF) [27], gray level cooccurrence matrix (GLCM) [28], hu invariant moment (huMoment) [29] and histogram of directional gradient (HOG) [30].

The illustration and parameters of different feature extraction methods are depicted in Table 4. Here the ovr SVM classifier with the RBF kernel function was established for the experiment. To avoid overfitting, a grid search method with 5-fold cross-validation was performed by taking 10 values separately at C ∈ [1, 100] and γ ∈ [10^-5, 10] to obtain the best classification results.

Table 4
Illustration & parameters of different feature extraction methods

Method Illustration &parameter

GHSF Calculate 5 histogram features of mean, variance, energy, entropy value and contrast of gray level.

GLCM Calculate a total of 24 features for 4 angles to extract texture features.

huMoment Extract 7 features to filter the object profile.

HOG 9 bins, each cell contains 32 × 32 pixels, each block contains 4 × 4 cells, L2 normalization

BOSF K=200 (the speed dataset), K = 100 (the fan dataset)

Method	Illustration &parameter
GHSF	Calculate 5 histogram features of mean, variance, energy, entropy value and contrast of gray level.
GLCM	Calculate a total of 24 features for 4 angles to extract texture features.
huMoment	Extract 7 features to filter the object profile.
HOG	9 bins, each cell contains 32 × 32 pixels, each block contains 4 × 4 cells, L2 normalization
BOSF	K=200 (the speed dataset), K = 100 (the fan dataset)

In Fig. 3, it can be seen that HOG and BOSF perform well, with scores above 0.95 for all the four evaluation indicators, as they utilize the local gray-level gradient information of the thermal images, while the other three methods perform poorly, with scores below 0.9 for all the four evaluation indicators. GHSF performs the worst as it only uses the global gray-level information of the thermal images. It is worth noting that BOSF has all scored close to 1. This is because the proposed method can further extract useful information from the preliminary fault feature matrix of each thermal image while reducing the feature dimensionality and better describing the thermal fault information of the components in the thermal image of the board, which basically does not lead to misdiagnosis and underdiagnosis.

Fig. 3

Comparison of the scores of different feature extraction methods (a) The speed dataset (b) The fan dataset.

4.5 Comparison of different classifiers

To prove that the multi-classification SVM model has stronger generalization capability, experiments are performed in conjunction with the BOSF fault feature extraction method. It was compared with two classifiers, error back propagation neural network (BP) [31] and Naive Bayesian (NB) [32].

The illustration and parameters of different classification methods are shown in Table 5. Here, the BOSF method was chosen for feature extraction, using the optimal parameters from Section 4.3.

Table 5
Illustration & parameters of different classifiers

Method Illustration &parameter

BP 2 hidden layers, 100 neurons and 50 neurons respectively, Maximum iteration 500

NB Gaussian Naive Bayesian

SVM “ovr” mode, RBF kernel function, C = 12.0,γ=0.000215 (the speed dataset)C = 1.0,γ= 0.1668 (the fan dataset)

Method	Illustration &parameter
BP	2 hidden layers, 100 neurons and 50 neurons respectively, Maximum iteration 500
NB	Gaussian Naive Bayesian
SVM	“ovr” mode, RBF kernel function, C = 12.0,γ=0.000215 (the speed dataset)C = 1.0,γ= 0.1668 (the fan dataset)

In Fig. 4, it can be seen that when BOSF is applied for feature extraction, the three classifiers score above 0.91 for the four evaluation indicators, which indicates that the component fault features extracted by BOSF have strong robustness. It is worth noting that the multi-classification SVM model in this paper has the best performance with all the four evaluation indicators higher than 0.995, while the other two have poor performance with all the four evaluation indicators lower than 0.95, and the BP classifier is the worst. It can be seen that the multi-classification SVM model used in this paper is more generalizable and more accurate than the other two classifiers.

Fig. 4

Comparison of the scores of different classifiers (a) The speed dataset (b) The fan dataset.

4.6 Comparison of different diagnostic methods

In order to verify the advantages of the proposed method over the traditional deep learning method under small sample datasets, BOSF-SVM and Resnet18 [33] were compared in experiments under the same conditions of dataset division, as shown in Table 6 and Fig. 5. Also, the potential of the proposed method is shown by verifying its performance relative to the state-of-the-art computer vision-based diagnostic methods for circuit boards. The competitors include SVM+D-S [12], 2D-DWT+PCA+SVM [13] and a CNN-based DL model [34], as summarized in Table 7.

Table 6
Comparison of the results of different diagnostic methods

Dataset Method Precision Recall f1-score Accuracy Time (s)

speed Resnet18 0.9700 0.9689 0.9688 0.9595 12.3940

BOSF-SVM 0.9977 0.9976 0.9976 0.9976 0.0268

fan Resnet18 0.9726 0.9714 0.9711 0.9714 8.6522

BOSF-SVM 0.9906 0.9905 0.9904 0.9905 0.0279

Dataset	Method	Precision	Recall	f1-score	Accuracy	Time (s)
speed	Resnet18	0.9700	0.9689	0.9688	0.9595	12.3940
	BOSF-SVM	0.9977	0.9976	0.9976	0.9976	0.0268
fan	Resnet18	0.9726	0.9714	0.9711	0.9714	8.6522
	BOSF-SVM	0.9906	0.9905	0.9904	0.9905	0.0279

Fig. 5

The confusion matrix of different methods (a) Resnet18 on the speed dataset (b) BOSF-SVM on the speed dataset (c) Resnet18 on the fan dataset (d) BOSF-SVM on the fan dataset.

Table 7

Comparison of the results of computer vision-based diagnostic methods

Method	Fault states	Dataset	Accuracy
SVM+D-S [12]	8(8 fault states on 3 chips)	Onboard A31	94%
2D-DWT+PCA+SVM [13]	2(healthy, faulty)	Arduino Uno	97.6%
A CNN-based DL model [34]	2(normal, defective)	PCBA	96.6%
BOSF-SVM	6(6 fault states on chips and resistors)	speed	99.76%
	3(3 fault states on a chip)	fan	99.05%

The parameters of the BOSF-SVM method were the best parameters obtained in section 4.3 and section 4.5. The Resnet18 method used the tensorflow-gpu 2.6 deep learning framework, setting epoch to 70 and batch size to 8, selecting the Adadelta optimizer and choosing the cross-entropy loss function.

The best results are marked in bold in Table 6. When faced with the task of small sample datasets, although the accuracy of both methods is high on all four evaluation indicators, the proposed method is better than Resnet18 on both datasets. In terms of diagnostic accuracy, the method is at least 2% higher than Resnet18 in both datasets. In terms of diagnostic speed, the single image prediction time of the method is milliseconds in both datasets, much faster than Resnet18. Besides, it can be seen from the confusion matrices in Fig. 5 that the proposed method has few sample classification errors on both datasets. It can be seen in Table 7, the proposed method performs well under both circuit board thermal image datasets, outperforming the latest comparative literature methods under optimal parameters with accuracy above 99% and diagnostic speed under 0.03s.

5 Conclusion

In this work, a thermal imaging-based intelligent fault diagnosis approach BOSF-SVM has been presented. It first employs secondary manual extraction of features to provide a more precise description of the faulty components in the thermal images of the circuit boards. Secondly, the multi-classification SVM model has the advantages of targeting a small sample size, high generalization capability, fast speed, and adjustable parameters, and is well suited to the situation where it is difficult to obtain a large number of fault thermal image samples from the board. Combining the above two aspects, the proposed method effectively reduces misdiagnosis and underdiagnosis, making the fault diagnosis process simple and efficient. Using an intelligent method to automate circuit board fault diagnosis, it is of practical significance to improve maintenance efficiency and provide a strong guarantee for early recovery of equipment operation in actual industrial production.

As the circuit boards used in this paper is relatively simple, the fault features are obvious, so the experimental effect is relatively good. However, the actual circuit boards are complex and diverse, and their fault features are not easy to extract, coupled with the possible problem of unbalanced samples of various types, so the fault diagnosis model will be further optimized in the future to make it more suitable for practical application scenarios.

Footnotes

Acknowledgments

This work has been supported by National Natural Science Foundation of China (No. 52175379), Liaoning Provincial Science and Technology Department (No. 2022JH2/101300268), and Postgraduate Education Reform Project of Liaoning Province(2-4).

References

Gao

, Yang

and Jiang

, A novel incipient fault diagnosis method for analog circuits based on GMKL-SVM and wavelet fusion features, IEEE Transactions on Instrumentation and Measurement 70 (2020), 1–15.

, He

and Li

, Generative adversarial networks with comprehensive wavelet feature for fault diagnosis of analog circuits, IEEE Transactions on Instrumentation and Measurement 69(9) (2020), 6640–6650.

Yang

, Meng

and Wang

, Data-driven feature extraction for analog circuit fault diagnosis using 1-D convolutional neural network, IEEE Access 8 (2020), 18305–18315.

, Lu

and Gao

, Pipeline image diagnosis algorithm based on neural immune ensemble learning, International Journal of Pressure Vessels and Piping 189 (2021), 104249.

, Du

, Wan

, Wang

and Yu

, Rotating machinery fault diagnosis based on convolutional neural network and infrared thermal imaging, Chinese Journal of Aeronautics 33(2) (2020), 427–438.

Wang

, Dong

, Ren

, Wu

, Guo

, Zhuang...

and Xie

, Automatic fault diagnosis of infrared insulator images based on image instance segmentation and temperature analysis, IEEE Transactions on Instrumentation and Measurement 69(8) (2020), 5345–5355.

Zhu

, Tang

, Zhang

, Chen

and Miao

, Infrared and visible image fusion based on edge-preserving and attention generative adversarial network, Journal of Infrared and Millimeter Waves 40(5) (2021), 696.

Chen

, Chen

, Xu

, Yang

and He

, Nv-Net: Efficient infrared image segmentation with convolutional neural networks in the low illumination environment, Infrared Physics & Technology 105 (2020), 103184.

Min

, Gu

, Li

and Yang

, Non-rigid infrared and visible image registration by enhanced affine transformation, Pattern Recognition 106 (2020), 107377.

10.

, Chen

, Fang

and Zhu

, Fault detection of on-board devices based on infrared thermal imaging, Laser and Infrared 48(5) (2018), 579–584.

11.

Wang

, Li

, Wu

and Zeng

, Research of aviation circuit board fault diagnosis and prediction based on infrared technology, Comput Simul 34(07) (2017), 89–93.

12.

Hao

and Jia

, Research on fault mode diagnosis of airborne circuit board based on infrared images, Infrared Technology 41(3) (2019), 273–278.

13.

Mehra

, Thermal Imaging-Based Fault Diagnosis of Electronics Circuit Boards, In Advances in Energy Technology (pp. 111–121), Springer, Singapore, (2022).

14.

Ahmed

, Ali

M.U.

, Hussain

S.J.

, Zafar

and Al Hasani

, Visual Vocabulary Based Photovoltaic Health Monitoring System Using Infrared Thermography, IEEE Access 10 (2022), 14409–14417.

15.

Kour

and Ünver

H.M.

, VisDroid: Android malware classification based on local and global image features, bag of visual words and machine learning techniques, Neural Computing and Applications 33(8) (2021), 3133–3153.

16.

, Hui

and Yong

, Pattern recognition of partial discharge in the presence of noise based on speeded up robust features, Transactions of China Electrotechnical Society 37(3) (2022), 775–785.

17.

Arora

, Dubey

A.K.

, Jaffery

Z.A.

and Rocha

, Bag of feature and support vector machine based early diagnosis of skin cancer, Neural Computing and Applications (2020), 1–8.

18.

Hou

, Zhao

and Qu

, The ability of infrared thermal imaging technology to diagnose printed circuit board faults, Infrared Technology 6 (1996), 37–41.

19.

Bay

, Ess

, Tuytelaars

and Van

, Gool, Speeded-up robust features (SURF), Computer Vision and Image Understanding 110(3) (2008), 346–359.

20.

Vlahović

and Djurovic

, Robust tracking of moving objects using thermal camera and speeded up robust features descriptor, International Journal of Adaptive Control and Signal Processing 35(4) (2021), 549–566.

21.

Xie

, Li

and Zhao

, Strain measurement based on speeded-up robust feature algorithm applied to microimages from a smartphone-based microscope, Sensors 20(10) (2020), 2805.

22.

Zhang

, Zhao

and Chen

, Application of migration image registration algorithm based on improved SURF in remote sensing image mosaic, IEEE Access 8 (2020), 163637–163645.

23.

Csurka

, Dance

, Fan

, Willamowski

and Bray

, Visual categorization with bags of keypoints, In Workshop on statistical learning in computer vision, ECCV (Vol. 1, No. 1-22, pp. 1–2), (2004).

24.

Cortes

and Vapnik

, Support-vector networks, Machine Learning 20(3) (1995), 273–297.

25.

Huang

, Lei

, Li

and Gu

, Damage identification of bridge structures considering temperature variations-based SVM and MFO, Journal of Aerospace Engineering 34(2) (2021), 04020113.

26.

Huang

, Zhang

, Li

, Wang

, Ma

, Wang

and Lv

, Classification of cervical biopsy images based on LASSO and EL-SVM, IEEE Access 8 (2020), 24219–24228.

27.

Tang

, Kong

and Wu

, Review of surface defect detection based on machine vision, Journal of Image and Graphics 22(12) (2017), 1640–1663.

28.

Mohanty

, Rup

, Dash

, Majhi

and Swamy

M.N.S.

, Digital mammogram classification using 2D-BDWT and GLCM features with FOA-based feature selection approach, Neural Computing and Applications 32(11) (2020), 7029–7043.

29.

Fredo

A.J.

, Abilash

R.S.

, Femi

, Raja

N.S.M.

and Kumar

C.S.

, Characterization of global and local damages in composite images using geometrical and Fourier-Hu moment–based shape descriptors, Journal of Testing and Evaluation 49(2) (2019), 796–813.

30.

Dalal

and Triggs

, (2005, June). Histograms of oriented gradients for human detection, In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05) (Vol. 1, pp. 886–893), Ieee.

31.

Wang

, Wu

and Li

, Fault identification of rolling bearings based on improved EMD decomposition method and BP neural network, In 2022 International Conference on Big Data, Information and Computer Network (BDICN) (pp. 650–655), IEEE, (2022).

32.

Niazi

K.A.K.

, Akhtar

, Khan

H.A.

, Yang

and Athar

, Hotspot diagnosis for solar photovoltaic modules using a Naive Bayes classifier, Solar Energy 190 (2019), 34–43.

33.

, Zhang

, Ren

and Sun

, Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778), (2016).

34.

Jeon

, Yoo

and Kim

S.W.

, A Contactless PCBA Defect Detection Method: Convolutional Neural Networks With Thermographic Images, IEEE Transactions on Components, Packaging and Manufacturing Technology 12(3) (2022), 489–501.

BOSF-SVM: A thermal image-based fault diagnosis method of circuit boards

Abstract

Keywords

1 Introduction

2.1 SURF algorithm

2.2 BOF model

2.3 SVM classifier

3 A thermal images-based fault diagnosis method: BOSF-SVM

3.2 Image preprocessing

3.3 BOSF feature extraction

4.1 Introduction of data sources

Table 2 Fan dataset division State code Thermal images Training set Test set Fault classes 0 30 70 ICbreak 1 30 70 ICshort 2 30 70 ICnormal

Footnotes

Acknowledgments

References

Table 2
Fan dataset division

State code Thermal images Training set Test set Fault classes

0 30 70 ICbreak

1 30 70 ICshort

2 30 70 ICnormal