Research of immune clone algorithm applying in the LED lamp automatic lock screw workstation

Abstract

A new automatic lock screw machine for LED lamp has been researched in the paper, which is based on the immune clone algorithm. Firstly, the robot automatic lock screw system has been setup. Vision sensor captured LED lampshade images in real-time. Median filtering operation has been performed to the captured images. On this basis, a new image segmentation method based on the immune clone algorithm has been researched. The antibody and antigen of LED lampshade image have been defined. The immune affinity function has been setup at the same time. Several artificial immune operators have been designed, which can be used to obtain the optimum image segmentation threshold value in real time. The final results showed that the optimum image threshold value can been obtained after less than ten times iterative computation. Compared with the traditional Ostu method, the new method can reduce the disturbance information around the threaded hole, which made basis for threaded hole recognition and location.

Keywords

Automatic screw locking machine robot station machine vision immune clone algorithm

1. Introduction

During the manufacture procession, locking screw technology can realize the connection between different parts by the engagement of screw and threaded hole, which is very important for assembly step. In the field of electric appliances manufacturing, electronic production manufacturing, furniture home appliances manufacturing, the locking screw technology is still performed by humans. Worker carries screw, puts it into the hole, then carries the electric screw driver, presses the electric screw driver button. And the locking between the screw and threaded hole can be realized. The traditional locking method is lower efficiency, and the accuracy of screw driving can’t be confirmed. Therefore, the realization of automatic locking screw is very important [1].

American and other Europe countries have developed the automatic locking screw machine firstly. During the 1950s, America developed the orbital typed automatic locking screw machine. The machine could pass the arranged screw to the locking screw mechanism through the pathway. Then the start button could be pressed and the screw could be automatically locked by the machine. The assembly efficiency was very high. However, the volume of the whole equipment was very huge, and structure of the machine was very complex, which wasn’t convenient for installment and placement. With the successfully development of the first industry robot in United States, robot technology has been applied for automatic locking screw in the field of assembly. The articulated typed locking screw robot has firstly developed in German, which has the intelligent judgement function [2].

In recent years, the economy of China has developed rapidly. The need of automatic locking screw equipment is great demanded in the area of Chinese Pearl River Delta and Yangtze River Delta. Nowadays, the common automatic locking screw equipment are the rectangular coordinate typed automatic locking screw machine, the four-axis locking screw robot and the six-axis joint typed locking robot. The program modes of the machines are teaching programming. Before using, the robot should be demonstrated and programmed,which is very time costing. Nowadays, the machine vision and artificial intelligent technology have developed rapidly. The working efficiency of the locking screw robot can be greatly improved by the application of these latest technologies. For example, with the merit of untouched, high efficiency and rapid computing capability of machine vision, the assembly quality can be greatly improved for the locking screw robot [3, 4, 5, 6].

In the paper, a new automatic lock screw method for LED lamp has been researched, which is based on machine vision. Vision sensor captures LED lampshade images and sent to industry computer. Image median filtering and segmentation operations have been performed. With the help of merits of immune algorithm, such as adaptivity, randomness, concurrency and so on, the optimum segmentation threshold of LED lamp screw hole image can be obtain. The coordination of screw hole can be calculated in real time and sent to industry robot. Compared with the traditional automatic locking screw ways, the new method is less time crossing and the position accuracy for screw hole is higher, which has became the trend of automatic screw locking machine in the future.

2. The self-developing automatic locking screw system fro LED lamp

The automatic locking screw system fro LED has been developed, which is shown in Fig. 1. The whole system contains ABB robot, light source, vision camera, industry control computer, and so on. The locking screw machine, CCD camera and light source have been installed on the end of ABB robot.

During working procession, when industry control computer received the auto locking screw command, ABB robot moved to the position which was above the LED lamp. The vision sensor captured LED lamp image and sent to industry control computer. Based on the image procession algorithm on the computer, the position of screw hole can be calculated. These position information was then sent to ABB robot, and ABB robot moved to the screw holes and locked the screws successively. Then the automatic of locking screw procession can be realized, the working flow is shown in Fig. 2.

Figure 1.

The self-developed automatic locking screw system.

Figure 2.

Procession of automatic locking screw.

3. Recognition algorithm for LED lamp screw hole based on the immune-clone algorithm

3.1 Capture and pretreament of lamp screw hole image

Based on the setup automatic locking screw system showed in Fig. 1, the LED lamp images can be captured by the CCD sensor, which is fixed on the robot clamp. Then the captured images have been sent to industry computer. In order to reduce the image procession time, a 530 $\times$ 500 rectangular region around screw hole is chosen to be the procession district, which is shown in Fig. 3.

Figure 3.

Lamp screw hole image.

In order to filter the uneven bright, reflecting light, and other uncertain disturbance, the median filter operator has been performed to the image. The image operation is actually one kinds of sorting filter based on the statistics. For one point ( $i, j$ ) in original figure, the filter made the sorting median to be new output pixel value around the point district. The template of median filtering is 3 $\times$ 3 in the paper. For example, the image pixel value of 8 neighbourhood of point ( $i, j$ ) is 12, 18, 18, 11, 23, 22, 13, 25, 118. The sorting results is 11, 12, 13, 18, 18, 22, 23, 25, 118. So the median pixel value is made to be the output pixel value, which is 18. However, the character of lamp screw hole can be reduce at the same time. And the gray enhancement operation is performed at the same time, which can be enhance the contrast between the lamp screw hole and its around district and the image restoration result is shown in Fig. 4.

Figure 4.

Image restoration result of screw hole image.

3.2 Recognition of screw hole image based on the immune clone algorithm

The artificial immune algorithm is one kind of bionic intelligent algorithm, which is inspired by the somatic cell and net theory of biological immunity system. By imitating the defence mechanism of biology immunity system, the algorithm offers several evolution operations, such as noise tolerance, unsupervised learning, self-organization, memorization, and so on. The algorithm also has the merit of classifier, neural network and machine inference. In recent years, the immune algorithm has been widely applied in the field of optimal computation, automatic control, and so on [7, 8, 9, 10, 11, 12].

At present, most research achievement of immune optimal computing is based on the clonal selection theory, which proposed by Burnet [13]. The theory thinks that the production of antibody can be divided into two stages. Before stimulated by antigen, living organism contains a large of cell group, which is made up by variety of antibody. The Embodied information is formed by the aeon evolution process of living organism. When stimulated by antigen, the higher affinity degree antibody group will has the priority to reproduce. And high frequency gene mutation is performed during the reproduce procession of antibody, so the clone operation is caught by the reproducing. Then some cloned cells are differentiated to be plasma cells and huge of antibodies can be produced to eliminate antigens. Others is to be memorized cells to joint the second immunoreaction. The clone selection theory describes the characters of antibodies simulated by antigens, the self-adapting immune response, and so on.

During the life cycle time, immune system will stimulate by same antigen cells. The original immune response to antigen is originally performed by little of B cells. And the generated antibodies have different affinity degree. In the original respond period, the higher affinity degree antibodies will be retained to form sub-population to confront intrusion of the same antigens. So the second response is performed, so the efficiency of immune response can be accelerated. The clone selection mechanism is the basic theory of immune optimal computation. The clone selection method CLONALG proposed by de Castro is the classical immune algorithm [14]. Before that time, Weinland, Forrest and Fukuda have also simulated the immune selection mechanism on other different perspectives, and the researched achievements have been applied to the fields of optimal computation, patter recognition, and so on [15, 16, 17].

In the paper, the immune algorithm has been researched in the application of LED lamp screw images, which is used to compute the optimal image segmentation threshold values [18, 19, 20, 21, 22, 23, 24, 25].

(1) Code and decode of lamp screw hole image

The captured LED screw hole images can be coded by binary. The gray levels of images are 0 to 255. Each image gray level contains 8 bits binary character string. The original character string can be randomly produced by computer. The decoded method is shown in the following formula.

$\displaystyle y=a_{8}\times 2^{7}+a_{7}\times 2^{6}+\ldots+a_{1}\times 2^{0}$ (1)

In the formula, $y$ is the gray level of LED lamp screw hole image. $a_{i}$ is the value at the $i$ position of the character string, which is 0 or 1.

(2) Affinity function of the LED lamp screw hole image

Supposed the gray threshold value of LED lamp screw hole image is $k$ . $C_{0}$ is the pixel set, which contains the [0, 1, …, $k-1$ ] gray levels. And $C_{1}$ is the pixel set, which contains the [ $k$ , $k+1$ , …, $L-1$ ] gray levels. $C_{0}$ is defined to be antigen, and $C_{1}$ is defined to be antibody. The inter class variance function is determined to be the affinity function of LED lamp screw hole image, which is shown in Eq. (3.2). The parameters of formula can be shown as following, and the parameter $n$ is the total pixel number of the pixel set.

$\displaystyle\sigma_{B}^{2}=\omega_{0}(\mu_{0}-\mu_{T})^{2}+\omega_{1}(\mu_{1}% -\mu_{T})^{2}$ $\displaystyle\left\{{{\begin{array}[]{l}{\omega_{0}=\sum\limits_{q=0}^{k-1}{p_% {q}(r_{q})}}\\ {\omega_{1}=\sum\limits_{q=k}^{L-1}{p_{q}(r_{q})}}\\ {\mu_{0}=\left.\sum\limits_{q=0}^{k-1}qp_{q}(r_{q})\right/\omega_{0}}\\ {\mu_{1}=\left.\sum\limits_{q=k}^{L-1}qp_{q}(r_{q})\right/\omega_{1}}\\ {\mu_{T}=\sum\limits_{q=0}^{L-1}{qp_{q}(r_{q})}}\\ {p_{q}(r_{q})=\frac{n_{q}}{n},q=0,1,2,\ldots,L-1}\\ \end{array}}}\right.$ (2)

(3) Antibody screening

The antibody screening procession is performed by the relative affinity degree between antibody and antigen. The higher affinity degree of antibody, the more antibody of new generation can be produced. The relative affinity degree can be determined by Eq. (3).

$\displaystyle p_{i}=\frac{f(d_{i})}{\sum\limits_{j=1}^{N}{f(d_{j})}}$ (3)

In the formula, $p_{i}$ is the the relative affinity degree of antibody. $d_{i}$ stand for antibody individuals. $f(d_{i})$ is the affinity between antibody and antigen.

(4) Immune genetics operator

Biology immune system has many antibody genetics operators. The artificial immune operators have been constructed by simulating the running mechanism of biology immune system.

1) Clone operator

The operator chooses the higher affinity antibody to clone according the relative affinity degree of antibody individuals. The new generating antibody individuals join into the antibody of next generation, raise up seed and take part in the genetic operation of next generation. The purpose of clone operation is to make sure that the excellent antibody individuals can generate more descendants, whose affinity is bigger. And so the parents’ better character can be passed on.

2) Crossover operator

The operator chooses random pair of antibody individuals, exchanges certain part of the antibodies. The crossover position is randomly determined. By the crossover operation, new antibody can be produced, which is shown in Fig. 5. So the gene of antibody individuals can be greatly improved, and search ability of the algorithm can be enhanced.

Figure 5.

Crossover operation of antibody.

3) Variation operator

Variation operation firstly chooses a antibody individual, and inverses one bit of the antibody by certain variation rate, which is 0 to 1 or 1 to 0. The procession is shown in Fig. 6. In order to improve the diversity of the new antibody, the variation rate can be increased. The high frequency variation can boost the ability of the algorithm for searching the optimal value, which can avoid the algorithm to be trapped into local optimal.

Figure 6.

Variation operator.

4) Memory operator

After the iterative computations have finished, the judgement has been performed to determine weather the best affinity of current generation antibody and antigen is higher than the last generation. If the answer is yes, the optimum antibody of current generation is then determined to be the memorized antibody group and joint the next iteration.

5) Inject vaccine operation

Suppose the antibody group is $c=(x_{1},x_{2},\ldots,x_{n})$ . The operation randomly chooses certain antibody individual from c and perform the injection vaccine to it. The inject vaccine operation is to modify the gene value at certain position of the antibody. Vaccine roots from the prior knowledge,so the operation can import the knowledge and experience of human. The antibody can obtain a higher affinity degree at high probability, and the convergence rate of algorithm can become faster.

4. Recognition of LED lamp screw hole image based on immune clone algorithm

Based on the setup automatic locking screw system shown in Fig. 1, LED lamp images can be captured in real time. The moving speed of assembly robot is 1.5 m/s, and the image sample time 40 ms. When capturing the LED lamp images, robot moves to the position just above the screw hole. Vision sensor captured images and sent to industry computer, which is fixed on the end of robot. The captured lamp image and procession district is shown in Fig. 3. Median filtering operation is firstly performed to the images. The artificial immune algorithm is constructed, and the optimal segmentation threshold value can be obtained. Then image segmentation operation is caught out.

The flow of artificial immune algorithm is shown in Fig. 7. The total individuals of antibody group in every generation is set to be 1000. Every antibody individual contains 8 gene bits, which stands for the 256 gray levels of LED lamp screw image. Variation rate of antibody is set to be 0.9. The original antibody group of artificial immune algorithm is produced by computer randomly. The affinity between antibody and antigen is calculated by Eq. (3.2). If the calculated affinity values is higher, the iteration computing would be terminated, and the result calculated by Eq. (1) can be obtained and used to be image segmentation threshold value. However, if the calculated affinity values is lower. Several immune operators would be performed, which are the immune clone, immune crossover, immune variation, immune memorization and inject vaccinate operations. Then the antibody group is refreshed, and the new iteration computing would be started,and the computing procedure is shown in Fig. 8.

Figure 7.

Flow of immune algorithm.

Figure 8.

Computing procedure of artificial immune algorithm.

Due to the abundant immune operators, the researched algorithm has many merits and characters,which are adaptivity, randomness, concurrency, and so on. For example, the crossover and variation operators can enhance the population diversity, which can avoid the local optimum. The memory and inject vaccine operators can accelerate the convergence speed of algorithm. From the computing procedure shown in Fig. 8, it can be found that the optimum segmentation threshold can be obtained after less than 10 generations’ iteration. The computing speed of researched algorithm is faster.

Figure 9.

Image segmentation results by immune algorithm and Ostu method.

The image segmentation results by the researched immune algorithm and Ostu method have been shown in Fig. 9. From the Fig. 9a, it can be found that the inner and outer edges of screw hole have been extracted by the proposed method. The isolated points disturbance around inner and outer screw holes have become less, which is very convenient to calculate the coordinate for the screw hole.

However, in Fig. 9b, the Ostu method can also extract the screw hole edge basically. But the edge of outer ring for screw hole was disconnected and discontinuous. The inner ring of screw hole missed too much edge. More isolated points appear around the hole edge, which has made more difficult to calculate the screw hole coordinate. So the LED lamp screw hole image after segmentation based on the immune algorithm seemed better than the traditional Ostu method.

5. Conclusion

This paper presented a new method for LED lamp automatic lock screwing,which is based on the immune clone selection algorithm. LED lamp locking screw system has been developed independently. The vision sensor has been fixed at the end of robot, which can be used to capture images in real time and send the images to industry computer.

A new image segmentation method has been researched, which is based on the immune artificial algorithm. By simulating the working mechanism of biology immune system between antibody and antigen, the frame of artificial immune algorithm has been constructed, which is combined to the traditional image segmentation operation. Then the optimum threshold value of image can be obtained.

Several testing experiments have been performed. The results showed that the developed immune algorithm has better performance, it can obtain the optimum segmentation threshold after about 7 times’ iteration. Compared to the traditional Ostu method, the immune clone algorithm can extract the inner and outer rings edge of screw hole. The isolated points disturbance around inner and outer screw holes have become less,which is very convenient to calculate the coordinate for the screw hole. However, the Ostu method can also the screw hole edge basically. But the edge of outer ring for screw hole extracted by the traditional Ostu method was disconnected and discontinuous. And more isolated points appear around the hole edge, which has made more difficult to calculate the screw hole coordinate.

Footnotes

Acknowledgments

The authors acknowledge the major nature science project of Guangdong colleges and universities (No. 2023ZDZX3092), Dongguan social development of science and technology key project (No. 20231800939952), Dongguan Sci-tech Commissioner Project of 2021 – Research and development of intelligent flexible production line (No. 20211800500042), intelligent terminal and intelligent manufacturing special project of Dongguan Polytechnic: Platform and team project (No. ZXA003), intelligent terminal and intelligent manufacturing special project of Dongguan Polytechnic (No. ZXD202307), Dongguan Science and Technology Ombudsman Project in 2022 (No. 20221800500812), Guangxi Key Lab of Manufacturing System & Advanced Manufacturing Technologyproject (No.22-035-4S018).

References

Frustaci

Spagnolo

Perri

Cocorullo

, et al. Robust and high-performance machine vision system for automatic quality inspection in assembly processes. Sensors. 2022; 22(8): 2839.

Fei

Xiaozhong

Jiawei

Lihui

. Design of mobile phone automatic assembly system based on machine vision. Journal of Physics: Conference Series. 2022; 32(1): 314-321.

Bingyi

Xiancheng

Junhua

Chuhua

Mengyao

. A novel non-contact measurement method of ball screw thread profile detection based on machine vision. Measurement Science Review. 2023; 23(1): 47-54.

Xiaobo

Yukun

Ying

Chengyi

, et al. Vision-aided brush alignment assembly system for precision conductive slip rings. Machines. 2022; 10(5): 393-408.

Hang

Changying

Ruijian

. Measurement methods of 3D shape of large-scale complex surfaces based on computer vision: A review. Measurement. 2022; 197(1): 211-224.

Ruizhi

Nguyen

Park

Yoon

. Automated picking-sorting system for assembling components in an IKEA chair based on the robotic vision system. International Journal of Computer Integrated Manufacturing. 2022; 35(6): 583-597.

Zhengyi

Wangwang

Yalun

. Symmetric uncertainty based decomposition multi-objective immune algorithm for feature selection. Swarm and Evolutionary Computation. 2023; 78(3): 86-107.

Bahashwan

Al Tuwairqi

. Modeling the effect of external computers and removable devices on a computer network with heterogeneous immunity. International Journal of Differential Equations. 2021; 43(5): 31-47.

Cohen

Efroni

Atlan

. Immune Computation and COVID-19 Mortality: A Rationale for IVIg. Critical Reviews in Immunology. 2020; 40(3): 195-203.

10.

Yukun

Yanmang

Yan

Bei

Shurui

. Improved immune algorithm for sudden cardiac death first aid drones site selection. International Journal of Medical Informatics. 2023; 173(3): 105025.

11.

Ting

Lin

Jie

. Systematic integration of machine learning algorithms to develop immune escape-related signatures to improve clinical outcomes in lung adenocarcinoma patients. Frontiers in Immunology. 2023; 175(11): 86-101.

12.

Gong

. Improved immune computation for high-precision face recognition. Soft Computing. 2017; 21(20): 5989-5999.

13.

Ji Chang

MyungKi

SangHun

DongKuk

. Improved Immune Algorithm Combined with Steepest Descent Method for Optimal Design of IPMSM for FCEV Traction Motor. Energies. 2021; 14(13): 3904-3915.

14.

De Castro

Von Zuben

. Learning and optimization using the clonal selection principle. IEEE Transaction on Evolutionary Computation. 2002; 6(3): 239-251.

15.

Peixin

Wei

Lingling

Guanhua

, et al. Deep learning predicts immune checkpoint inhibitor-related pneumonitis from pretreatment computed tomography images. Frontiers in Physiology. 2022; 13(2): 978222.

16.

Forrest

Javornik

Smith

, et al. Using genetic algorithms to explore pattern recognition in the immune system. Evolutionary Computation. 1993; 1(3): 191-211.

17.

Lan

ZhenHui

LiXu

Xin

. Development and validation of a computed tomography-based immune ecosystem diversity index as an imaging biomarker in non-small cell lung cancer. European Radiology. 2022; 32(12): 8726-8736.

18.

Hongdan

Yan

Bing

Zhigang

. Research on detection mechanism of vessel abnormal behavior based on immune genetic spectral clustering algorithm. Ocean Engineering. 2022; 266(5): 5-25.

19.

Yunliang

Xuyang

Zhen

Xiongtao

, et al. An adaptive immune-following algorithm for intelligent optimal schedule of multiregional agricultural machinery. International Journal of Intelligent Systems. 2022; 37(11): 9404-9423.

20.

Jianhua

Zhiheng

. Artificial immune algorithm-sparrow search algorithm and its application in network intrusion detection. Journal of Intelligent and Fuzzy Systems. 2022; 43(4): 5001-5011.

21.

Wenhui

. Optimal planning of electric vehicle charging stations considering user satisfaction and charging convenience. Energies. 2022; 15(14): 5027-5037.

22.

Yuejun

Liyong

Yong

. Thyroid extraction based on artificial immune network with firefly algorithm in single photon emission computed tomography image. Journal of Medical Imaging and Health Informatics. 2020; 10(3): 614-619.

23.

Jing

Hang

Desheng

. Modified immune evolutionary algorithm for medical data clustering and feature extraction under cloud computing environment. Journal of Healthcare Engineering. 2020; 125(7): 1051394.

24.

Cohen Irun

Efroni

. The immune system computes the state of the body: Crowd wisdom, machine learning, and immune cell reference repertoires help manage inflammation. Frontiers in Immunology. 2019; 10(6): 10-24.

25.

Yabin

Chenghao

Jin

. Immune scheduling network based method for task scheduling in decentralized fog computing. Wireless Communications and Mobile Computing. 2018; 58(9): 1-8.