The method of repairing OHE damaged ancient painted murals based on machine vision

Abstract

Traditional mural repair methods only observe the texture of murals when segmenting the repair area, but ignore the extraction of a mural damage data, resulting in incomplete damage crack information. For this reason, the method of repairing the damaged murals based on machine vision is studied. Using machine vision, it can get two-dimensional image of a mural, preprocess the image, extract the damaged data of a mural, and then divide the repair area and repair degree index. According to different types of damage, it can choose the corresponding repair methods to achieve the repair of damaged mural. The results show: Compared with the OPTICS-based unsupervised method and the machine vision for orchard navigation method, the number of repair points and repair cracks extracted by the proposed method is more than that of the two traditional methods, which can more accurately and comprehensively extract the repair information of murals.

Keywords

Machine vision painted mural repair method damaged image painting grouting

1. Introduction

An ancient painted mural is an art work painted on the wall, which shows rich historical and cultural information. It has a history of more than a thousand years. It is a national treasure and a world heritage. However, due to the long history of ancient painted mural, the wall is not conducive to the long-term preservation of the mural. At the same time, affected by natural disasters and human factors, it has caused a variety of damage, such as fading, discoloration, shedding, etc., so it is important and necessary to repair the ancient painted murals. In order to repair the integrity of the mural and achieve the required visual effect, the damaged parts of the mural are repaired and filled, or the objects in the image are removed.

Therefore, a method of repairing the damaged ancient painted murals based on machine vision is proposed. In this method, machine vision is to use machine instead of human eye to measure and judge. Through machine vision products, such as CMOS and CCD image pickup devices, the ancient color murals captured are transformed into image signals, which are transmitted to a special image processing system. According to the information of pixel distribution, brightness, color, etc., they are transformed into digital signals. These signals are carried out various operations by the image system to obtain the damage characteristics of ancient painted murals, and then control the on-site equipment actions according to the discrimination results. Machine vision has an inestimable value in detecting the damage of ancient painted murals. While repairing, it can permanently preserve the image information of the painted murals, demonstrate the process and the reality of the damaged murals, and provide a reference for the damage repair of ancient painted murals.

2. Similar works

At present, many researchers at home and abroad have put forward many methods for repairing the damage of ancient painted murals, which can be divided into the following two categories: Reference [1] proposed an image damage repair method based on local differentiation and mutation. The repair process is to spread the known information along the structural information in the painted murals, and through the principle of variation, the partial differential formula and variation model are equivalently deduced. The main idea of the model is to first establish functional, consider the existing part and prior model in the painted mural, establish the repaired functional according to these two items, and according to the theory of variational functional, transform the restoration of ancient painted mural into an energy functional problem to find the extreme value. The edge information around the area to be repaired is used to complete the restoration. Reference [2] proposed a mural damage repair method based on transform domain. Firstly, the painted murals are transformed to the transformation domain, and then the characteristics of the transformation domain are used to repair the damaged murals. If there is an over complete dictionary, a few atoms in the dictionary can be used to represent the signals effectively, calculate the sparse coding of the painted murals, and then use the corresponding over complete dictionary and sparse coding, to complete the restoration and reconstruction of the painted murals, so as to achieve the restoration of the damaged areas in the murals. However, due to the variety of damaged forms of ancient painted murals, if they are repaired according to the above two traditional methods, there will inevitably be problems such as the distortion of information and the difficulty of long-term preservation of painted murals. Because the repair work is irreversible operation, there are certain risks. Reference [3] proposes an adaptive restoration method of damaged image lost area based on texture synthesis. The damaged image lost area is texture segmented, the edge energy feature of damaged image is extracted, and the adaptive texture density quantization estimation is completed. The texture vector of the lost region is calculated to quantify the super-pixel visual features of the region. The priority repair region is calculated according to the texture characteristics based on the Criminisi algorithm, and the reliability of the image is continuously updated to repair the lost region, so as to complete the adaptive repair of the lost region of the damaged image. Reference [4] proposes an improved sparse representation method based on similar matching block groups. The method of combining color information and cosine distance is used to define image block matching criteria to obtain matching block groups with more similar structural change trend in the target neighborhood; Then, in the process of sparse reconstruction, the known information and estimated unknown information are considered at the same time, and different weights are added to the sparse coefficients by using the matching degree between similar blocks and target blocks, so as to enhance the ability of filtering matching blocks and reduce the phenomenon of texture blur; Finally, according to the structural sparsity, the sample block size is adaptively determined in the areas with different structural complexity to reduce the error propagation phenomenon in the process of image restoration. However, the above methods only observe the texture of murals, but ignore the extraction of mural damage data, resulting in incomplete damage crack information.

3. The design of the method of repairing the damaged ancient painted murals based on machine vision

3.1 Obtaining mural image based on machine vision

In order to obtain high-quality ancient painted mural image, the machine vision detection technology is used to simulate the human visual function. At the same time, it is necessary to accurately perceive the mural position and control the process of capturing the painted mural image with the camera. The image acquisition process includes four controls: machine vision detection, camera control, camera photographing, segmentation and positioning of the painted mural image manufacturing process.

After the machine vision detects the damage of the painted mural, it sends the location information of the damage to the camera control unit, which controls the camera shutter to open and takes photos. The image is transferred to the image segmentation and positioning unit for processing, the position of the image in the whole mural is located, and the position deviation between the image center and the whole mural center is calculated and feedback to the camera control unit. According to the position deviation of this time, the camera control unit adjusts the opening time of the next camera shutter, realizes the accurate capture of the color mural by the camera, and obtains the two-dimensional image of all positions of the ancient color mural.

3.2 Mural image preprocessing

3.2.1 Image denoising

Because there is usually noise in the damaged mural image, which will affect the extraction effect of the boundary information in the damaged area of the image, so the key work of preprocessing is to denoise the image. Using anisotropic diffusion to deal with the noise of a mural image, the two-dimensional image is regarded as a thermal field, and each pixel is regarded as a thermal flow. According to the relationship between the surrounding pixels and the current pixels, it can decide whether to diffuse to the surrounding thermal field. The difference between the current pixel and the neighboring pixel is relatively large, which means that the neighboring pixel is likely to be the boundary, so the current pixel will not be in this direction. If the image above is extended, the boundary of the damaged area will be preserved [5]. If the mural image is $I$ , the number of iterations is $t$ , the coefficient of thermal conductivity in four directions is $\textit{cN},\textit{cS},\textit{cE},\textit{cW}$ , the iterative coefficient is $t$ , and the image is $x$ in the transverse direction and $y$ in the longitudinal direction, then the main iterative formula is:

$\displaystyle I_{{t}+1}=I_{t}+\left({cN_{{x,y}}\nabla_{N}\left({I_{t}}\right)+% cS_{{x,y}}\nabla_{S}\left({I_{t}}\right)+cE_{{x,y}}\nabla_{E}\left({I_{t}}% \right)+cW_{{x,y}}\nabla_{W}\left({I_{t}}\right)}\right)$ (1)

Four different divergence formulas are used to calculate the partial derivative of the current pixel in four directions. The formula is as follows:

$\displaystyle\left\{{\begin{array}[]{l}\nabla_{N}\left({I_{{x,y}}}\right)=I_{{% x,y-1}}{-}I_{{x,y}}\\ \nabla_{S}\left({I_{{x,y}}}\right)=I_{{x,y-1}}{-}I_{{x,y}}\\ \nabla_{E}\left({I_{{x,y}}}\right)=I_{{x,y-1}}{-}I_{{x,y}}\\ \nabla_{W}\left({I_{{x,y}}}\right)=I_{{x,y-1}}{-}I_{{x,y}}\\ \end{array}}\right.$ (2)

Anisotropic diffusion selects PM mode, whose basic idea is to smooth mural image in the region of scale space, reduce and smooth the boundary between region and region, and the minimum coefficient is generally on the boundary. If the area coefficient is ${k}$ , the formula determining the smoothness is:

$\displaystyle\left\{{\begin{array}[]{l}cN_{{x,y}}=\text{exp}\left({{-}\left\|{% \nabla_{N}\left(I\right)}\right\|^{2}/{k}^{2}}\right)\\ cS_{{x,y}}=\text{exp}\left({{-}\left\|{\nabla_{S}\left(I\right)}\right\|^{2}/{% k}^{2}}\right)\\ cE_{{x,y}}=\text{exp}\left({{-}\left\|{\nabla_{E}\left(I\right)}\right\|^{2}/{% k}^{2}}\right)\\ cW_{{x,y}}=\text{exp}\left({{-}\left\|{\nabla_{W}\left(I\right)}\right\|^{2}/{% k}^{2}}\right)\\ \end{array}}\right.$ (3)

In Eq. (3), the divergence operator and gradient operator are exp and $\nabla$ , and the diffusion formula is the anisotropic diffusion formula of discrete type, which is applied to the pretreatment of a mural image [6]. If the constant controlling the diffusion degree is $\lambda$ , the discrete sampling of the current image is $I_{P}^{t}$ , that is, the corresponding coordinate position of the pixel is $P$ , that is, the adjacent area to $P$ is $\eta_{P}$ , and that of the damaged pixel is ${q}$ , then the anisotropic diffusion formula is:

$\displaystyle I_{P}^{{t}+1}=I_{P}^{t}+\frac{\lambda}{\left|{\eta_{P}}\right|}% \sum\limits_{{q}\in\eta_{P}}{c\left({\nabla I_{P,{q}}^{t}}\right)\nabla I_{P,{% q}}^{t}}$ (4)

A monotone decreasing function of gradient is obtained. Anisotropic diffusion denoising filter is used for the input image to be repaired to enhance the texture information of the inner structure of the image and enhance the boundary blur caused by expansion.

3.2.2 Corrected radiation

In order to collect the damaged data of a mural image, radiation correction preprocessing is needed. The method of correction is to collect dark current data and standard white board data through instrument, so as to realize reflectivity inversion. By scanning standard white board image with reflectivity of 100%, we can get all white calibration image data. The dark current data is the all black calibration image data with zero reflectivity of the sealed light box under the same lighting conditions [7]. If the reflectivity of dark current data is dark, the reflectivity of white board data is white, the reflectivity of image acquisition data is data, and the reflectivity of corrected data is ref, then the radiation correction formula is:

$\displaystyle\text{ref}=\frac{\text{data}-\text{dark}}{\text{white}-\text{dark}}$ (5)

The collected data are transformed by MNF to eliminate the correlation between the bands, and then the first several bands with high radiation are selected for the inverse transformation of MNF through statistical characteristic value curve after MNF transformation. After MNF transformation, the characteristic components of the image characteristic value curve are mainly concentrated in the first 10 bands with high radiation, and the first 10 characteristic bands are selected for the inverse transformation of MNF to transform the spectrum of the image. After the information is converted back to the original data space, the image processed by the minimum radiation separation forward transformation and inverse transformation can not only retain the spectral information of a mural, but also suppress the radiation in the image.

3.3 Extraction of the damaged data of a mural image

3.3.1 Extraction of the damage data in paint layer

After image denoising, radiation correction and other preprocessing, on the basis of this data, the ROI average spectrum of different pigment categories is directly selected from the image, the ASCII file is saved, and the average spectrum information is established. Then, the end-element spectrum of mural pigment is extracted and the end-element spectrum information is established by using the convex cone algorithm of continuous maximum angle, and two different spectral data are used as training samples. The method of spectral angle mapping is used to extract the damaged data of pigment layer, and then the extraction results are analyzed and the accuracy is evaluated [8]. The extraction process is as shown in Fig. 1.

Figure 1.

Flow chart of damage data extraction.

The damaged data exists in a pixel in the form of mixed spectrum. Through the image of MNF inverse transformation, the region of interest at different positions is selected to obtain the spectral curve of multiple positions of each substance. The average spectral curve of each ROI is obtained by the average value, and the average spectrum is established as the basis for determining the damaged mural image. At the same time, the continuous maximum angle convex cone is used to extract the end-element spectrum and abundance image. Let the end element of the spectrum be $H$ , the band index and the pixel index be $m, n$ respectively, the index from 1 to the maximum end-element ${N}$ be ${k,j}$ respectively, the matrix of the end-element spectrum be ${R}$ , and the abundance matrix of the end-element ${j}$ to the end-element ${k}$ in each pixel be ${A}$ , then the mathematical formula adopted by the maximum angle convex cone is:

$\displaystyle sH\left({{m,n}}\right){=}\sum\limits_{k}^{N}{{R}\left({{m,k}}% \right)}{A}\left({{k,j}}\right)$ (6)

According to the calculation results, the end-element spectrum curves of different pigments are obtained and the end-element spectrum is established. Finally, using the method of spectral angle mapping classification, taking the mean spectrum and the end-point spectrum as training samples, the MNF inverse transformation is classified to extract the damage information of mural paint layer.

3.3.2 Extraction of texture damage data

The features that can reflect the texture attributes of the image are extracted, to get the texture damage data, as the main basis to identify the damage of the painted murals [9]. According to the damage characteristics of ancient painted murals, the multi-scale characteristics of wavelet transform are used to quickly extract the texture features of any scale, so as to describe the texture operator of the local texture spatial structure of the image. The local binary mode of mural image is determined by taking a certain pixel in the image as the center and comparing with its surrounding pixel points. The local image is shown in Fig. 2.

Figure 2.

Pixel node set of local binary mode.

Through wavelet transform, the texture image is decomposed into point sets of different scales in the figure, and the high-frequency details of the horizontal, vertical and diagonal directions of the texture are obtained. On the low-frequency and high-frequency information of each scale, the texture statistical characteristics are calculated respectively, which can fully describe the texture information and improve the identification performance of texture features [10]. If the gray value of the center pixel of a domain is ${g}_{c}$ , the number of image pixels is $P$ , the radius is $R$ , the gray value of the pixel is ${g}_{p}$ , and the binary function is ${s}$ , then the calculation formula of the local binary mode $\textit{LBP}_{P,R}$ is:

$\displaystyle\textit{LBP}_{P,R}{=}\sum\limits_{P}^{P-1}{{s}\left({{g}_{P}-{g}_% {c}}\right)2^{P}}$ (7)

By extracting rotation related features, the rotation independent texture features in the pattern are obtained. The spatial scale and rotation attributes are considered synthetically. The spatial scale of wavelet transform is used to eliminate the influence of spatial scale and rotation, and the gray scale and rotation attributes are considered at the same time. In order to eliminate the influence of rotation, the texture features independent of rotation are:

$\displaystyle\textit{LBP}_{P,R}^{r}{=\min}\left\{{\textit{ROR}\left({\textit{% LBP}_{P,R},{i}}\right)\left|{{i=}0,1,\ldots,P{-}1}\right.}\right\}$ (8)

In Eq. (8), ROR means to move right circularly for ${i}$ sites, to describe the local texture characteristics of the image. The statistical information can be used to characterize the texture features of the image. As part of the local binary pattern depicts the basic properties of the texture, the pattern that depicts the texture-based properties is regarded as a unified mode, each unified pattern as a texture feature, while all non-unified modes are classified as one. In the aspect of texture feature extraction, in the ${x}$ and ${y}$ directions of mural image, one-dimensional wavelet transform is performed respectively to decompose the image into four independent subbands on all scales, which correspond to the low-frequency information, vertical high-frequency information, horizontal high-frequency information and high-frequency edge information of image texture [11]. The result of decomposition depends on the selected wavelet type. Four independent subbands are set as LL, LH, HL, HH and mural image as $I$ . The formula of decomposition process is as shown in Fig. 3.

$\displaystyle\left\{{\begin{array}[]{l}\textit{LL}=\left[{L_{x}*\left[{L_{y}*I% }\right]_{\downarrow 2}}\right]_{\downarrow 2}\left({{x,y}}\right)\\ \textit{LH}{=}\left[{L_{x}*\left[{H_{y}*I}\right]_{\downarrow 2}}\right]_{% \downarrow 2}\left({{x,y}}\right)\\ \textit{HL}{=}\left[{H_{x}*\left[{L_{y}*I}\right]_{\downarrow 2}}\right]_{% \downarrow 2}\left({{x,y}}\right)\\ \textit{HH}{=}\left[{H_{x}*\left[{H_{y}*I}\right]_{\downarrow 2}}\right]_{% \downarrow 2}\left({{x,y}}\right)\\ \end{array}}\right.$ (9)

In the formula, $*$ is convolution operation, $\downarrow 2$ is down sampling operation, $H$ and $L$ are image high-pass filter and image low-pass filter respectively. To continue the above operation for low-frequency sub image LL, multi-level two-dimensional wavelet transform of image can be realized, and the schematic diagram of two-level discrete wavelet decomposition of image is as follows:

Figure 3.

Schematic diagram of two-level discrete wavelet decomposition of mural image.

The low-frequency sub image of each scale contains the main information of the image texture under the current scale. Therefore, the histogram of the original image and the low-frequency sub image of each scale is combined to form a larger histogram, which combines the average histograms of LH and HL subband images at different scales, so that the texture features of the local binary pattern of the image can be obtained, and the texture damage data of the mural can be obtained

3.4 Identification of broken murals

Based on the extracted damaged data of mural image, the damaged murals are identified. In order to render the coating effect of mural surface, ancient color murals often brush transparent coating on the mural surface, which plays the role of protecting murals [12, 13]. However, after the influence of environmental changes and human factors, the materials used to make these murals will be damaged. The common damages include caustic soda, crazing, armor rising, hollowing, smoke, microorganism and pigment layer falling off, etc. The damage information of ancient color murals is analyzed, the damage types and marking diagrams of murals are summarized, and a database for identifying the damage of murals is established, as shown in Table 1.

Table 1
Damage identification database

Name	Damage type	Illustration	Explanation
Craze			The tiny network crack on the surface of mural
Exanthema			The phenomenon that the base color layer, pigment layer or surface coating of murals show scaly curling
Bullous goiter			The phenomenon of bubbling, breaking and curling up of the pigment layer and the base color layer of murals
Pulverization			The phenomenon that the gelatinous material in the pigment layer of mural is aged and falls off in the form of powder or fine particles
Pigment layer falling off			The phenomenon that the pigment layer is separated from the base color layer or the ground layer
Punctate abscission			The phenomenon that the base color layer is off the ground or the pigment layer is off the base color layer in the form of dots
Scabby exfoliation			The phenomenon of abscission of the scar like protuberance formed by scab disease
Fracturing			Dislocation and cracking of murals
Scratching			Marks left by external forces
Scribble			Artificial writing or daubing on the screen

Table 1, continued
Name	Damage type	Illustration	Explanation
Smoky			Traces of the picture being inspected or stained by incense
Salt frost			Salt crystal on screen surface
Crisp alkali			The loose state caused by the action of soluble salt
Hollowing			The phenomenon that the ground floor is partially separated from the support and connected with the support
External damage			All kinds of damage to murals caused by external forces
Microbial damage			The harm of microbe breeding to mural, such as mildew, fungus, etc.

The database shown in Table 1 is used as the recognition standard for the damage of ancient painted murals.

3.5 Division of mural restoration area

The paper divides the damaged areas of different damaged murals, and transforms the problem of region division into the problem of energy function minimization and optimization based on image segmentation. The energy function is divided into two parts. The first part is the likelihood energy, which reflects the similarity information between pixels and foreground and background in the image to be segmented. The second part is the prior energy, which reflects the correlation information between pixels in the image to be segmented, which has nothing to do with the outside world and can be directly defined from the image [14]. Firstly, the repair area is pre segmented, and the mean shift segmentation algorithm is adopted. The main body of kernel function is assumed to be the normalization constant ${k}\left(\bullet\right)$ , and the bandwidth parameter is ${h}$ . For the given ${n}$ sample points in ${d}$ -dimensional space ${R}^{d}$ , if ${x}_{i},{i=}1,\ldots,{n}$ , then the kernel density estimation function ${f}\left({x}\right)$ of multidimensional variable is:

$\displaystyle{f}\left({x}\right){=}\frac{1}{{nh}^{d}}\sum\limits_{{i=}1}^{n}{% \left\|{\frac{{x-x}_{i}}{{h}}}\right\|^{2}}$ (10)

To find the local stable points in the feature space, it has ${f}\left({x}\right){=0}$ at these local stable points. According to the adaptive gradient rising method of the mean shift algorithm, it tends to converge to the local extreme point. To analyze the low-density value area, the difference between the weighted mean values is calculated, to obtain that the mean shift pace is relatively large, but close to the local extreme point. Its mean shift pace is relatively small. The ${d}$ -dimensional input vector generated by the input image is the filtered ${d}$ -dimensional vector. Let the center of the kernel window be ${x}$ , the space domain part of the eigenvector be ${x}^{s}$ , the range domain part of the eigenvector be ${x}^{r}$ , the kernel function be ${k}_{{h}_{s},{h}_{r}}\left({x}\right)$ , the bandwidth of the kernel be ${h}_{s},{h}_{r}$ , the normalization constant be $T$ , the main body of the kernel function be ${k}\left({x}\right)$ , then the calculation formula of the multi-dimensional variable in the mean shift is:

$\displaystyle{k}_{{h}_{s},{h}_{r}}\left({x}\right){=}\frac{T}{{h}_{s}^{2}{h}_{% r}^{2}}{k}\left({\left\|{\frac{{x}^{s}}{{h}_{s}}}\right\|^{2}}\right){k}\left(% {\left\|{\frac{{x}^{r}}{{h}_{r}}}\right\|^{2}}\right)$ (11)

To carry out the clustering algorithm in the mixed domain, all the ${x}$ whose spatial distance is less than ${h}_{s}$ and color feature difference is less than ${h}_{r}$ is aggregated together to form different categories. Labeling each pixel according to the category, the number of pixels in each damaged area is calculated [15]. If the characteristic value of the training data is ${w}$ , the target weight is ${w}_{0}$ , and the initial value of the core window center is ${y}$ , then the calculation formula of the number of pixels in the damaged area is:

$\displaystyle{y}\left({{x;w}}\right){=}\sum\limits_{{i=}1}^{n}{{wk}\left({{x,x% }_{i}}\right){+w}_{0}}$ (12)

All areas with pixels less than ${h}_{s}$ are removed to get the final segmentation image of the damaged area.

3.6 Calculation of mural restoration index

By evaluating the damage degree of the mural and calculating the restoration index of the ancient painted mural, it can choose the restoration technology and determine whether the previous restoration measures are still effective or whether they will cause new damage. The relevant evaluation indexes that affect the current conditions of murals are selected, and the number of diseases, disease area and other factors is used to directly reflect the severity of the damage. Since there is more than one damage type on a mural, but each type of area is not large, four indexes, i.e. comprehensive disease area, quantity, density and aesthetic degree, are proposed to give weight, and then the color mural repair index is obtained as a quantitative method for determining repair status and repair degree [16].

The index of damaged area is selected to measure the scope of the mural damage. The statistics of the scope vary from region to region. The repair index is the ratio of the damaged area to the total area of the mural, which is used to measure the general degree of the damage. The statistics take a single mural or a specific research area as the calculation scope. The higher the damage rate of the mural in a unit area is, the greater the degree of repair is needed. If the damaged area of each mural is $S_{i}\left({{i=}1,\ldots,{m}}\right)$ and the area of mural is $S$ , the damage rate index DRI is:

$\displaystyle\textit{DRI}{=}\frac{\sum\nolimits_{{i=}1}^{m}{S_{i}}}{S}$ (13)

The damage density index is selected, the density state is described according to the damage type, the texture of the mural itself and the damage scope of the damage type, and the damage density is introduced, which is used to analyze the spatial characteristics of the damage and describe the degree of the fragmentation of the mural. The greater the index value is, the greater the degree of the mural to be repaired is [17, 18]. If the number of damaged murals is $N$ and the area of murals is $S$ , the damage density index $D I$ is:

$\displaystyle\textit{DI}{=}\frac{N}{S}$ (14)

The damage boundary density index is selected to calculate the length of the damage boundary in unit area, which is the basic index to analyze the damage shape, reflecting the complexity of the damage boundary characteristics. The more complex the damage boundary is, the more impact the integrity of mural repair is. If the total length of the damaged boundary is $E$ and the total length of the mural boundary is $A$ , then the density index BDI of the damaged boundary is:

$\displaystyle\textit{BDI}{=}\frac{E}{A}$ (15)

The index of mural fragmentation is selected, to evaluate the degree of mural fragmentation, represent the interference degree of environment and human influence, and judge the degree of mural to be repaired by visual characteristics. If the number of broken murals is $M$ and the average rea of broken murals is $A_{p}$ , the index FI of broken murals is:

$\displaystyle\textit{FI}=\frac{M}{A_{p}}$ (16)

The extracted mural damage information is quantified into a conceptual numerical form to express the damage occurrence status in different levels, and the level of repair degree is defined. Multiple regression analysis method is used to interpret the multiple index factors of single damage and evaluate the impact on the ancient painted mural [19].

3.7 Painting and grouting to repair murals

According to the type of mural damage, the restoration is carried out in the mural area. The restoration degree is mainly based on the restoration index. The restoration methods are as follows. When the pigment layer is powdered, bubbled or armoured, spray paint shall be used to make the mixture fall on the pigment layer of the mural. First, the pigment layer is reinforced, and then the mural with cotton ball row is pressed to ensure compaction. The cotton ball is made of thin and white satin wrapped with absorbent cotton. The diameter of the cotton ball is about 5 cm. The surface is sprayed with adhesive, and then the mural surface with a volume ratio of 4:11. 1.5% polyvinyl alcohol aqueous solution and 1% polyvinyl acetate emulsion mixture is sprayed [20]. Finally, the picture with a soft rubber roller is rolled. When the surface of the mural is sprayed with a mixture of paint and reaches 70% dryness, the white silk is spread on the mural, and slowly weighed and pressed with a soft rubber roller. When pressing with sulfur, even force is used to prevent the scale marks on the mural or stick the pigment layer on the white silk.

When the mural changes color and fades, the shadow line method is used to repair it. Watercolor is used to make the filling transparent and reversible. If there is any mistake, it is washed off with clear water. Firstly, the light background color of part of the coating is filled in. Then, a straight line is drawn from light to deep. The vertical line with light color similar to the background color is drawn for the first time, and the pattern color is deleted for the second time. The color matching should be more important than the first time, and the line drawing for the third time should be more important than the second time. It needs to be repeated many times so that the mural can see a complete picture from a distance, and the lines on the repaired mural will not be mixed with the original one [21].

Table 2
Parameters of machine vision instrument

Parameter name of spectrometer	Data parameters of spectrometer	Spectrometer parameter name	Spectrometer data parameters
Spectral range	350–2500 nm	Spectral range	400–1000 nm
spectral resolution	350–1000 nm @ 3 nm; 1001–2500 nm @ 8 nm	Spectral width	0.6 nm
Size and weight	12.7 $\times$ 35.6 $\times$ 29.2 cm; 5.44 kg	Weight	1.85 kg
Sampling interval	One4 nm @ 350–1000 nm; 2 nm @ 1001–2500 nm	Band number	1040
Number of channels	2151	Spectral resolution	2.6 nm

Table 3

Comparison results of statistical damage information

Region	Extraction data	Methods of this paper	Method in reference [1]	Method in reference [2]
1	Damaged quantity	223	180	167
	Length of damage boundary (mm)	13402	7350	8362
	Damaged area (mm ${}^{2}$ )	44469	16550	18123
2	Damaged quantity	89	74	62
	Length of damage boundary (mm)	3888	2654	2134
	Damaged area (mm ${}^{2}$ )	5027	4432	4006
3	Damaged quantity	178	154	143
	Length of damage boundary (mm)	10280	8825	7933
	Damaged area (mm ${}^{2}$ )	25632	19265	15660
4	Damaged quantity	16	11	6
	Length of damage boundary (mm)	686	476	369
	Damaged area (mm ${}^{2}$ )	1530	1379	1248

When the mural is empty, the pressure grouting and the combination of grouting and riveting are used. Using the gauze bar with epoxy resin or polyvinyl acetate emulsion fills the hollow part between the ground and the rock mass from the breach of the wall. Appropriate pressure is evenly applied to make the wall close to the rock mass and stick to the wall after the adhesive is solidified. The wall of the wall is filled with 15% polyvinyl acetate emulsion sand and lime paste, and a polyvinyl acetate emulsion is added to the front edge of the painting to make the bonding more firm. Meanwhile, the moisture in the lime paste is prevented from infiltrating to the surface of the murals, leaving traces. The filling material shall be the material similar to the ground layer material, and it shall be noted that the material shall not be too hard, shall not react with the raw material, shall not have too strong strength, shall not have too large shrinkage, and the filling degree shall be lower than the pigment layer [22]. When the mural surface is polluted, the physical method should be used to the greatest extent, and the chemical method should be used to the least, so as to destroy the combination of foreign matters and mural materials through mechanical action, to achieve the purpose of cleaning up the pollution. So far, the design of the method of repairing the damaged ancient painted murals based on machine vision has been realized [23].

4. Experimental verification

In order to verify the application performance of the method of repairing the damaged ancient painted murals based on machine vision, the proposed method is compared with the methods in reference [1] and reference [2] to verify whether the designed method can solve the problem of incomplete repair of the cracks and textures of the ancient painted murals.

Figure 4.

Comparison results of extracting and repairing cracks.

4.1 Experiment preparation

In this mural painting, there are many defective areas, and they are distributed in different texture patterns. There are not only complex texture patterns, but also simple linear structure, which affects the presentation of the complete information of the mural. First of all, three repair methods are used to judge the damage of the mural. In this paper, the image preprocessing and information extraction are carried out for the mural obtained by machine vision. The instrument used by machine vision is the portable ground object spectrometer and hyperspectral imaging spectrometer produced by ASD company. The point image data of the mural and the image data of the mural are obtained respectively. The relevant parameters of the instrument are as shown in Table 2.

Read in the mural image to be repaired, preprocess the read-in mural, set the pixel block and spectral information parameters in the mural image, and demarcate the damaged area of the mural, which is divided into four areas on average. After the preprocessing, file the mural image, so as to display and preview, sort and organize the results. The three repair methods count the damaged data of the mural image, and the comparison results are shown in Table 3.

Through the statistics and comparison of the data in the table, it can be seen that the number of mural image damage information extracted by this method is more than that of reference [1] and reference [2].

4.2 Division of fracture repair area

Three methods are used to divide the cracks in the murals. The results are as shown in Fig. 4.

It can be seen from the comparison results of Fig. 4 that compared with the methods in reference [1] and reference [2], the repair method in this paper can extract the cracks of the mural more accurately. The repair cracks cover more comprehensively in the mural. In the overall structure of the painted mural, the loss information block of the damaged area is the smallest. And the extraction time of the proposed method is 7.372 s, the extraction time of reference [1] and reference [2] are 21.472 s and 25.865 s respectively, showing that the proposed method shortens the extraction time of repairing cracks,

5. Conclusion

In order to improve the problem of incomplete damage crack information in traditional mural damage repair methods, the concept of replacing human eye detection with machine vision is proposed in this paper. Machine vision technology is used to extract the mural damage data and divide the mural repair area. According to the different types of damage, the corresponding repair methods can be selected to realize the repair of damaged murals. The comparison experiment shows that this paper makes the repair information in mural more comprehensive, can better judge the cracks to be repaired in the damaged area of mural, and improves the extraction accuracy of repair data. In addition, the extraction time of the three methods is compared. The extraction efficiency of this method is higher than that of the traditional method. However, the image of mural based on machine vision lacks adaptability, which makes it difficult for the parameters of detection and estimation to reach the exact point, thus affecting the restoration quality of ancient painted mural, making the overall color of the repaired mural lighter, and some parts appear unnatural phenomenon of restoration. The next work will improve the restoration method to improve the restoration accuracy and repair the mural according to the above shortcomings. The reconstructed image is clearer and more natural.

References

Sun

Wang

Chai

. OPTICS-based unsupervised method for flaking degree evaluation on the murals in Mogao Grottoes. Scientific Reports. 2018; 8(1): 1967-1974.

Josiah

Julie

Duke

. Machine vision for orchard navigation. Computers in Industry. 2018; 98: 165-171.

Liao

. Simulation of adaptive optimization and repair method for damaged image loss region. Computer Simulation. 2019; 36(6): 388-392.

Lou

Tang

Zhang

. Sparsity image inpainting algorithm based on similar patch group. Journal of Image and Graphics. 2019; 24(7): 1055-1066.

Madhavaraju

Saucedo-Samaniego

Löser

EspinozaMaldonado

Solari

Monreal

GrijalvaNoriega

JaquesAyala

. Detrital zircon record of Mesozoic volcanic arcs in the Lower Cretaceous Mural Limestone, northwestern Mexico. Geological Journal. 2018; 54(4): 2621-2645.

Elhagrassy

. Isolation and characterization of actinomycetes from Mural paintings of Snu-Sert-Ankh tomb, their antimicrobial activity, and their biodeterioration. Microbiological Research. 2018; 216: 47-55.

Giulia

Flavia

Martina

Doretta

Paolo

. Changes in biodeterioration patterns of mural paintings: Multi-temporal mapping for a preventive conservation strategy in the Crypt of the Original Sin (Matera, Italy). Journal of Cultural Heritage. 2019; 40: 59-68.

Morillas

Maguregui

Bastante

Huallparimachi

Marcaida

García-Florentino

Astete

Madariaga

. Characterization of the Inkaterra rock shelter paintings exposed to tropical climate (Machupicchu, Peru). Microchemical Journal. 2018; 137: 422-428.

Tamburini

Martin de Fonjaudran

Verri

Accorsi

Acocella

Zerbetto

Rava

Whittaker

Saunders

Cather

. New insights into the composition of Indian yellow and its use in a Rajasthani wall painting. Microchemical Journal. 2018; 137: 238-249.

10.

Ranalli

Zanardini

Rampazzi

Corti

Andreotti

Colombini

Bosch-Roig

Lustrato

Giantomassi

Zari

Virilli

. Onsite advanced biocleaning system for historical wall paintings using new agar-gauze bacteria gel. Journal of Applied Microbiology. 2019; 126(6): 1785-1796.

11.

Mateos

Cosano

Esquivel

Osuna

Jiménez-Sanchidrián

Ruiz

. Use of Raman microspectroscopy to characterize wallpaintings in Cerro de las Cabezas and the Roman villa of Priego de Cordoba (Spain). Vibrational Spectroscopy. 2018; 96: 143-149.

12.

Tatjana

. Buddhist wall paintings at Nako Monastery, North India: Changing of the technology throughout centuries. Studies in Conservation. 2018; 63(3): 171-188.

13.

Alzarok

Fletcher

Longstaff

. Survey of the current practices and challenges for vision systems in industrial robotic grasping and assembly applications. Advances in Industrial Engineering and Management. 2020; 9(1): 19-30.

14.

Tamburini

Martin de Fonjaudran

Verri

Accorsi

Acocella

Zerbetto

Rava

Whittaker

Saunders

Cather

. New insights into the composition of Indian yellow and its use in a Rajasthani wall painting. Microchemical Journal. 2018; 137: 238-249.

15.

Ziȩba-Palus

Kowalski

. The influence of the type of substrate on the possibility of spray paint identification for forensic purposes. Vibrational Spectroscopy. 2018; 95: 57-61.

16.

Al Huseini

Kasi

Shafaamri

Wonnie Ma

Subramaniam

. Study of the physical and electrochemical properties of hybrid paint system based on zinc-rich primer for mild steel protection. Pigment and Resin Technology. 2020; 49(1): 33-40.

17.

Gomes

Dionísio

Santiago Pozo-Antonio

Rivas

Rami

. Mechanical and laser cleaning of spray graffiti paints on a granite subjected to a SO 2-rich atmosphere. Construction and Building Materials. 2018; 188: 621-632.

18.

Wang

. Deep drainage detection system for inland vessels based on machine vision. Jordan Journal of Mechanical and Industrial Engineering. 2020; 14(1): 119-128.

19.

Grafia

Martini

Barbosa

. Spray process to styrene grafting onto polyethylene film surface for paintability enhancement. Progress in Organic Coatings. 2018; 117: 91-101.

20.

Ollier

Talle

Brisset

Bihan

Duerr

Lemmens

Goudemand

Robert

Hilbert

Buerstmayr

. Whitened kernel surface: A fast and reliable method for assessing Fusarium severity on cereal grains by digital picture analysis. Plant Breeding. 2019; 138(1): 69-81.

21.

Wang

Tian

Meo

Ciampa

. Image processing based quantitative damage evaluation in composites with long pulse thermography. NDT and E International. 2018; 99: 93-104.

22.

Navas

VMT

Buljac

Hild

Morgeneyer

Helfen

Bernacki

Bouchard

. A comparative study of image segmentation methods for micromechanical simulations of ductile damage. Computational Materials Science. 2019; 159: 43-65.

23.

Liu

Zheng

Huang

Fan

Wei

Yao

Emilia

Jain

. Design and implementation of intelligent tracking car based on machine vision. Journal of Intelligent and Fuzzy Systems. 2020; 38(5): 5799-5810.

The method of repairing OHE damaged ancient painted murals based on machine vision

Abstract

Keywords

1. Introduction

2. Similar works

3. The design of the method of repairing the damaged ancient painted murals based on machine vision

3.1 Obtaining mural image based on machine vision

3.2 Mural image preprocessing

3.2.1 Image denoising

3.3.1 Extraction of the damage data in paint layer

Table 1 Damage identification database

Table 2 Parameters of machine vision instrument

4.2 Division of fracture repair area

5. Conclusion

References

Table 1
Damage identification database

Table 2
Parameters of machine vision instrument