A comprehensive automatic labeling and repair strategy for cracks and peeling conditions of literary murals in ancient buildings

Abstract

To protect the historical and cultural heritage, the application of self-organizing mapping networks and genetic algorithms in the restoration of ancient architectural murals is studied. The results show that the average repair time for different types of mural paintings is less than 60 seconds, and the shortest repair time is only 17.81 seconds. The evaluation effect of the research model is good, and the comprehensive efficiency evaluation of the mural restoration work is improved by about 40.42%. The repair system has excellent performance, and the algorithm has high feasibility and effectiveness. The impact of restoring murals is substantial, and the extent of restoration is highly consequential for the restoration of ancient architectural murals.

Keywords

Ancient architectural murals self-organizing networks genetic algorithms automatic annotation restoration strategies

1 Introduction

Frescoes on ancient buildings embody the national culture and millennia of folk customs and style. They serve not only as historical artifacts but also as architectural painting art treasures. Murals are often painted on the walls or tops of buildings, and are often damaged by wind, sand, rain, and geological disasters due to changes in natural environment and geological conditions. The changes in the inherent structure of some murals and the aging of production materials have intensified the process of mural destruction over time [1 –3]. Under the influence of various factors, existing frescoes will inevitably develop cracks, peeling, mold, rust, and other ailments. To protect the historical and cultural heritage of ancient edifices, it is necessary to restore the ancient architectural frescoes. However, the traditional restoration method has its drawbacks, such as time-consuming, labor-intensive, and insufficient repair effect. Therefore, innovative digital techniques and intelligent strategies for frescoes restoration are urgently needed, which is the primary goal of this study.

With the progress of computer technology and image processing technology, promoting the digital construction of frescoes and the development of information technology, fresco restoration has provided a new research direction. Currently, fresco restoration primarily uses computer technology to digitally capture and restore the frescoes [4, 5]. The traditional restoration method for mural damage repair reflects the simultaneous repair of structure and texture, but the repair effect around the contour line of the mural is insufficient. The digital restoration technique focuses primarily on image restoration in visual conditions. The sample block-based restoration method can enhance restoration outcomes while simultaneously reducing restoration time, enhancing sample block restoration efficiency, and minimizing sample space redundancy.

Therefore, this study proposes a unique solution that repairs the structure of the image first and then labels any damage to the mural after restoration. This transforms the restoration process into a neural network optimization problem that achieves a complete connection of the information structure in the damaged area. As a result, the broken area is filled, and a high-quality restoration effect is achieved. The proposed method provides an effective solution to the issue of mural damage restoration and surpasses other existing methods in terms of both restoration process efficiency and quality.

The rest of the paper is organized as follows. Section 2 contains the literature review. Section 3 constructs a system of crack marking and falling off restoration of ancient building murals. Section 4 is the analysis of automatic marking and repair system for cracks and peeling of ancient architectural frescoes. Section 5 is the conclusion, which provides a comprehensive solution for automatic labeling and restoration of damaged murals in ancient buildings and verifies its application effect.

2 Related works

Various types of damage exist among the different structural levels of existing murals. Current research on mural conservation addresses the restoration of common defects found in murals. Tao et al. proposed a gated convolution-based defogging network for the renovation and comprehensive conservation of Dunhuang murals. A neural network with gated convolution was applied to restore the detached areas of the mural to ensure the integrity of the mural content. Second, the defogging nets were applied to improve the image quality to cope with the fading of the murals. Experimental results showed that after experimenting with the established Dunhuang mural dataset, the mural could be effectively restored to ensure the integrity of the mural content [6]. Although this method could effectively restore mural paintings and ensure the integrity of mural paintings, further research was needed to verify its effect in the face of large-scale and complex texture mural paintings. Purkait’s team, on the other hand, proposed a sample-based coherent texture synthesis technique to restore digital images of damaged parts of murals and paintings. And the proposed patch-based diffusion technique was combined with a high frequency generation technique that could achieve edge sharpening and denoising simultaneously. The experimental results and restoration results indicated that the method could handle a wide range of painting types with good restoration results [7]. However, because this technique relied on existing samples, its repair effectiveness was limited by the number and quality of samples. W Liu et al. proposed a sequence similarity detection-based restoration algorithm for ancient tomb murals to address the existing problems of fuzzy restoration and low pixel similarity matching in ancient tomb mural restoration. The cracks were separated out by a connected domain labeling algorithm and an open-close operation to complete edge threshold segmentation. The experimental results showed that the model could effectively repair the edges of ancient tomb murals and had a high degree of pixel similarity [8]. Nevertheless, this method consumed more computing resources and required high image preprocessing, so it might not be suitable for real-time restoration or large-scale application. While et al. proposed a complementary and microrefraction spectroscopy technique to stratigraphically analyze frescoes and identify fresco paints. The identification and analysis of the San Miguel Church in Argentina concluded that this technique could determine the materiality of the paintings and provide some theoretical basis for fresco restoration and conservation [9]. Madariaga’s team proposed the use of Raman spectroscopy and X-ray fluorescence portable equipment for the original pigments. New materials used in modern times, construction materials and decay products were characterized, and some collected samples were analyzed by laboratory techniques. The experimental results showed that the method could effectively determine the different elemental features on the murals with bacterial species [10]. JL W et al. investigated the effects of salt/water ratios on the appearance and rate of development of mural damage in non-destructive testing (NDT) methods. Nine kinds of broken fresco samples with salt/water ratio were made and placed in a constant temperature and humidity chamber for dry and wet cycle test. The experimental results showed that samples with a salt/water ratio of 1 : 10 did not suffer significant damage during the study time frame. The development speed of mural destruction was in time stages. The higher the saltwater ratio, the greater the risk of wall crack failure and the faster the development rate [11]. Y TAO’s team proposed a GD-Net-based gated convolution for the renovation and comprehensive protection of Dunhuang murals. The data set of Dunhuang mural paintings was established and tested, showing that the data set contained 1180 mural images of 290 and 112 caves in Mogao Grottoes. The experimental results proved the effectiveness and superiority of GD-Net [12].

The above results provide a physical basis for the restoration of mural paintings. But in the specific operation process, the accuracy of technical equipment and the technical proficiency of the operator may have an impact on the restoration effect.

Digital restoration techniques are primarily utilized for image restoration, now ubiquitous in medical imaging, automated monitoring, and restoring ancient artifacts. Beddad et al. proposed an efficient algorithm to enhance medical images corrupted by impulse noise. An enhanced nonlinear filter was used to remove low impulse noise density and high impulse noise density. After optimizing the proposed denoising algorithm and using it, the results indicated successful testing on various medical images with significantly reduced noise levels. Better peak signal-to-noise ratio results could be output and the algorithm running time was significantly reduced [13]. In the process of restoring images of ancient relics, CHen’s team proposed an adaptive restoration algorithm for Dunhuang murals based on an improved CDD model and an improved adaptive control strategy. The results of the empirical restoration of Dunhuang murals showed that the algorithm could solve the problems of unnatural transition edges and long restoration time of the CDD algorithm. It outperformed other algorithms in terms of PSNR and recovery time evaluation [14]. H Li, on the other hand, proposed a restoration refinement by dividing mural images based on local gradient features, which complemented image defects by extracting different pixel data and using a discrete difference algorithm. The experimental results showed that the method could satisfy the image continuity law, shorten the restoration time and recover the fresco cracks in an intelligent way [15]. Although these methods had achieved certain results, they were still powerless in the face of large-scale, complex texture and severely damaged fresco restoration.

To sum up, although there are a variety of restoration techniques for mural paintings, most of them are only suitable for straight line, curve and small regional scale restoration. When the damage area of the mural has rich texture and large damage degree, it can not play a good role. Therefore, the Self-Organizing Map (SOM) network and genetic algorithm are applied in the restoration of ancient architectural murals to automatically label and repair the cluster and layered murals. Even when the damage area of the murals has rich textures and large areas of damage, the optimal visual effect can still be conveyed and the historical and cultural information expressed by the murals can be restored.

3 Construction of crack marking and peeling repair system for frescoes of ancient buildings

3.1 Construction of mural painting disease labeling system for ancient buildings

The digital restoration and preservation of ancient architectural murals is a holistic process that safeguards thousands of years of human history and cultural heritage. To restore these murals, an initial process of mural damage labeling is essential. This provides mask images for repairing the damaged areas before the final restoration of ancient architectural murals can be accomplished. Cracks in ancient architectural murals can be divided into pre-cracks and post-cracks, with pre-cracks gradually developing into post-cracks after time and other factors. An example of mural damage is shown in Fig. 1.

Fig. 1

Sketch map of mural diseases.

The linear structural features and image grayscale processing combined with mathematical morphology can simplify the image data, enhance the crack shape features in the data and remove the irrelevant structures. Image grayscale processing has fast segmentation speed, less complex calculation and excellent chromaticity brightness level, which can meet the needs of fresco crack labeling and pre-processing [16]. To make the grayscale image of the mural more accurate and reasonable, the weighted average method is used for the color image, and the processing formula is shown in Equation (1).

$\begin{matrix} G (x, y) = 0.299 \times R (x, y) + 0.578 \\ \times (G (x, y) + 0.114 \times B (x, y) \end{matrix}$ (1)

In Equation (1), G (x, y) is the grayscale value of the weighted grayscale image at the corresponding coordinate. R (x, y) represents the R-component grayscale value at the corresponding coordinate. G (x, y) represents the G-component grayscale value at the corresponding coordinate. B (x, y) represents the B-component grayscale value at the corresponding coordinate. Image gradient detection is utilized to determine the rate of grayscale change surrounding the target boundary and edge, particularly in areas where significant changes in grayscale occur. The formula for calculating the gradient value of the pixel point (x, y) is shown in Equation (2). $G (x, y) = \sqrt{G_{x} (x, y)^{2} + G_{y} (x, y)^{2}}$ (2)

In Equation (2), G_x (x, y) is the gradient of the pixel point in the x direction. G_y (x, y) is the gradient of the pixel point on the defense line of y. The gradient direction of the pixel point is calculated as shown in Equation (3). $a (x, y) = arctan [\frac{G_{y} (x, y)}{G_{x} (x, y)}]$ (3)

The gradient direction indicates the fluctuation range of the two-dimensional discrete function of the digital image. If the gradient changes significantly, it indicates that the gray value varies too much within the region and may exist at the edge of the image. If the gradient changes slowly, it indicates that the gray value changes insignificantly and the image is flat in the corresponding region. And to highlight the highlighted areas, Morphological Gradient (MG) is used to mark the processing. The original mural image is set to be f (x, y) and the structural element is b (x, y), then the single-scale morphological gradient is calculated as shown in Equation (4). $G (f (x, y)) = (f (x, y) \oplus b (x, y)) - (f (x, y) \otimes b (x, y))$ (4)

In Equation (4), ⊕ represents the corrosion operator and ⊗ represents the expansion operator. The multi-scale morphological gradient is calculated as shown in Equation (5). $\begin{matrix} G_{n} (f (x, y)) = \frac{1}{n} \\ \sum_{i = 1}^{n} [(f (x, y) \oplus b_{i} (x, y)) - (f (x, y) \otimes b_{i} (x, y)) \otimes b_{i - 1} (x, y)] \end{matrix}$ (5)

In Equation (5), n is the multi-scale gradient. After conducting multi-scale morphological edge gradient detection to retrieve the base contour edge features of ancient architectural frescoes, the OTSU method is employed to adjust the threshold range adaptively, thus guaranteeing that the cracks are included in the target region. After adjusting for the cracks in the fresco, there still remains image noise. To address this issue and the discontinuity in noise, the study has implemented the connected marker rule to eliminate non-target point noise from the fresco image. When the pixel value of the target area of the image is 1px, the image area can be calculated by Equation (6). $A_{s} = \sum_{(x, y) \in s} f (x, y)$ (6)

In Equation (6), s indicates the connected area to be measured and f (x, y) is the pixel value. When the area of the connected domain is less than the set threshold, it is defined as the presence of noise to be removed. For another common issue affecting ancient architectural murals – mural peeling – this study analyzes the murals’ color characteristics in color space. The image is then enhanced through filtering and denoising, followed by the use of gradient transform and high hat transform to achieve the desired mural peeling target extraction. Among them, the median filtered point pixels in the process of image filtering after transforming the space can be expressed by Equation (7). $g (x, y) = med {f (i - x, j - y)} ((i, j) \in S)$ (7)

In Equation (7), the gray value of a point in the image is f (x, y). The filter window is set to M × N. (i, j) is the point inside the filter window. S is the region composed of points inside the filter window. For the scale enhancement of the image edges, the high hat transform is chosen to extract the mask region more flexibly and effectively. The formula for calculating the high hat transform is shown in Equation (8). $T_{hat} (f) = f - (f \circ b)$ (8)

In Equation (8), b is the structure element and f is the original image. After thresholding the original image to obtain the shedding edge grayscale map, the mask is internally filled and labeled after preserving the edge mask using the connected domain labeling algorithm.

3.2 Construction of crack disease repair system for ancient building frescoes

After the cracks of ancient architectural murals were automatically labeled, the study selected the SOM network algorithm for repair according to the comprehensive automatic labeling results. The SOM network performed adaptive learning and simulation by simulating the brain’s nervous system. The SOM network structure is shown in Fig. 2.

Fig. 2

SOM network structure.

SOM neural network is able to map high-dimensional data to two-dimensional space while maintaining the topology of the input data in the high-dimensional space. However, the traditional SOM image restoration method restores murals mainly by diffusion of the clustered and layered images, resulting in unsatisfactory restoration results. Therefore, the study of parallel iterative operation of the clustered layered images can effectively improve the restoration accuracy and restoration speed.

The improved self-organized mapping restoration process is shown in Fig. 3. For the fresco crack restoration process, the images are clustered and parallelized in layers after identifying the fresco crack range, and finally the restoration results are output after merging the multi-layer images. In the image clustering process, the color images are obtained and generated by RGB color space with the spatial coordinate system as shown in Fig. 4.

Fig. 3

Fresco disease repair process.

Fig. 4

Schematic diagram of color space.

The color space is generated with red, green and blue as the coordinate system, and the image information is obtained by light reflection acting on the system. In the RGB three-dimensional space, there exists a self-organizing network of the current input pattern x as n dimensional vector, and the input pattern x is input to the system to find the winning neuron. The finding process is shown in Equation (9). $d_{j} = \sqrt{\sum_{j = 1}^{m} {[x - w_{j}]}^{2}}$ (9)

In Equation (9), the number of nodes in the output layer is m × n. w_j is the weight vector of the j-th neuron node in the competitive layer. d_j is the Euclidean distance between the weight vector and the input pattern, i.e., the minimum winning node. For the weight adjustment authority, only the winning neuron has the right to adjust the weights as shown in Equation (10). $w_{j} (t + 1) = w_{j} (t) + η (t, N) [x (t) - w_{j} (t)]$ (10)

In Equation (10), η is the learning rate, given an initial value. η (t, N) is the topological relationship function between the running time t and the superior neighborhood. N_x(t) is the input pattern at the corresponding moment. w_j (t) is the weight vector at the corresponding moment. The hierarchical restoration for image parallelization, on the other hand, is an improved hierarchical processing of colorful RGB images. The iterative restoration sequence for broken pixels is schematically shown in Fig. 5.

Fig. 5

Iterative repair order.

In Fig. 5(a), when there is a broken pixel R, x is the unbroken pixel and y is the broken pixel. First, all broken pixels are searched and the broken pixel R is found whose neighboring pixels are all x. To mark R and confirm the layer where the pixel is located, and repair the broken pixel by iteratively calculating the broken pixel value. For the rest of the broken pixels, finding the broken pixel R whose neighboring pixels are not all x. The search and computation are repeated to complete the repair of all broken pixels. By using parallelized iterative SOM algorithm, instead of directly copying reference image pixels for repair, the pixel mean value is calculated iteratively. After clustering and layering the images, the model is computed in parallel to achieve high-speed finding and accurate repair.

3.3 Ancient building fresco shedding disease repair system construction

For the restoration of peeling frescoes of ancient buildings, the study extracted structural information from the peeling areas of frescoes based on the labeling of frescoes, and used Genetic Algorithm (GA) to connect the structural information and fill in the restoration in areas. For the extraction of structural information, the Canny edge detection algorithm was chosen. Firstly, the image was filtered by smoothing with Gaussian function and noise reduction. Secondly, the gradient of each pixel of the image was calculated to obtain the contour information. After extracting all the contour lines of the calibrated mural, the original structure information was analyzed and differentiated, which can be divided into “relevant information” and “irrelevant information” according to the different structure information. The “relevant information” refers to the different contour lines that are in contact with the detached area of the mural. The “irrelevant information” refers to the contour lines that do not exist in the original image and other contour lines that are not connected to the detached area. After processing the “irrelevant information”, the structure of the contour lines related to the peeling off of the mural can be obtained [17]. The formula for calculating the average brightness value in the basic features of the image around the damaged area is shown in Equation (11). $b = \sum_{i = 1}^{L} z_{i} p (z_{i}) = \sum_{i = 1}^{L} z_{i} \frac{z_{i}}{\sum_{i = 1}^{L} z_{i}}$ (11)

In Equation (11), L is the total number of pixels in the line pipe region. z_i indicates the gray value of the corresponding pixel in the region of interest. p (z_i) is the probability that the current luminance value occurs in the region. The average contrast formula is shown in Equation (12). $σ = \sqrt{\sum_{i = 1}^{L} (z_{i} - b)^{2} p (z_{i})}$ (12)

After extracting the relevant structural information and basic features of the peeling disease of the mural, a GA was used to optimize the structural information and perfect the smooth connection of the contour curves. To simplify calculations and accurately match the contour lines of shedding diseases, two optimization objectives were determined. The first objective assumed that there was at least one pair of matching related contour lines. The second objective required that the related contour lines must be connected within the damaged area. The GA objective function was designed using curve fitting based on the two objective assumptions to recover the structural information of the damaged area of the mural image. The degree of contour line matching was then determined by the curve smoothness together with the basic features of the image around the damaged area. For the matched contour lines, the curvature calculation formula at any point is shown in Equation (13). $K_{i} (x) = \frac{| y_{i}^{''} |}{(1 + y_{i}^{' 2})^{\frac{3}{2}}}$ (13)

In Equation (13), the contour line is set to be y = f (x), and x is any point on the contour line. The average value of the absolute value of the curvature rate of change is shown in Equation (14). $\bar{K} = \frac{1}{b - a} \int_{a}^{b} | \frac{{dK}_{x}}{dx} | dx$ (14)

In Equation (14), $\bar{K}$ is the mean value of the absolute value of the rate of change. Combining Equations. (13)-(14), the optimization objective can be expressed as an objective function, as shown in Equation (15). $g = \frac{β + \sum_{i = 1}^{M} D_{i}}{M}$ (15)

In Equation (15), β is the matching threshold parameter to balance the optimization target. M is the number of contour line matching pairs, and D_i is the matching degree of any pair of matching contour lines. When the optimal matching contour lines are obtained, the broken and undamaged areas of the detached area of the mural are divided in combination with the original contour base, and the broken area is filled with texture. The process of detachment repair and texture filling is shown in Fig. 6.

Fig. 6

Schematic diagram of filling and repairing.

In Fig. 6, the green area is the broken area. The contour lines connected with the green area are the broken area contour lines and the related contour lines connected with the breakage. The related contour lines of the unbroken region are extended and perfected with structural information. The mural image is divided into two parts for texture filling with structural information. In the process of partition texture filling, the filling priority calculation formula of the mural filling area is shown in Equation (16).

$P (p) = max {nS (p)}$ (16)

In Equation (16), nS (p) is the number of pixels in the unbroken area in the repair area. The number of repair blocks is determined by the number of pixels in the broken area. The formula for finding repair blocks with maximum fill priority is shown in Equation (17). $P (p \land) = {max}_{i = 1}^{M} {P (p)}$ (17)

After obtaining the repair block with the maximum fill priority, the best matching block is determined and the formula for determining it is shown in Equation (18). $ψ_{q \land} = min_{ψ_{p} \in D} {d (ψ_{p \land}, ψ_{q})}$ (18)

In Equation (18), d (ψ_p∧, ψ_q) is the sum of squared color value errors between two pixel blocks.

The system process of repairing mural cracks and falls off is shown in Fig. 7.

Fig. 7

Automatic annotation repair system process.

In the restoration system, the traditional SOM image restoration algorithm is improved, and the restoration speed and precision are improved through parallel iterative calculation, which shows a good restoration effect of ancient building murals. At the same time, for the lost part of the mural contour, GA mural shedding repair algorithm is used, and the algorithm combined with GA is used to repair the structure and texture information of the image respectively. Specifically, the optimal contour restoration scheme is found by curve fitting method to realize structural information restoration. On this basis, the texture information is repaired by region segmentation and filling.

4 Analysis of automatic marking and repair system for cracks and peeling of ancient architectural frescoes

4.1 Analysis of the effect of marking ancient architectural murals and the performance of the crack disease repair system

For the analysis of the annotation effect of frescoes of ancient buildings, the frescoes with diseases were selected for empirical annotation analysis. In this study, data preprocessing and image classification were implemented through Python language programming, and the Scikit-learn library was used to carry out pattern recognition and clustering with its powerful machine learning module. OpenCV and PIL were used for image reading, display and saving, as well as image filtering, threshold segmentation, edge detection and other pre-processing. Figure 8 shows the annotation results of frescoes with cracks.

Fig. 8

The labeling effect of mural diseases.

As shown in Fig. 8(a), there is an obvious crack disease and multiple shedding diseases in the mural. Figure 8(b) shows the crack mask and shedding mask during the labeling process, and Fig. 8(c) shows the labeling results. The labeling system established by the study can correctly mark the range of ancient architectural mural diseases, and the crack repair work and shedding repair work can be carried out normally. After marking the diseased areas of ancient architectural murals, the damaged areas marked by the identification are repaired with the primary goal of ensuring the historical and cultural values contained in the original murals. For the performance analysis of the disease restoration system, the restoration results were analyzed from both qualitative and quantitative evaluation perspectives. The qualitative analysis evaluated whether the restored images were natural and met the visual effect requirements from the perspective of visual effect. The quantitative analysis evaluated the similarity between the restored image and the original image by comparing them. In conducting the performance analysis, the root mean square error and peak signal-to-noise ratio were first selected to compare the performance gap between different restoration systems and the study system. The types of fresco lesions are shown in Table 1.

Table 1

Types of mural diseases

Number	Name	Disease type
Z1	Macaque Map	Single crack
Z2	Monk Map	Single crack
Z3	Bear Map	Multiple cracks
Z4	Color pattern	Multiple cracks
Z5	Buddhist image	Complex cracks
Z6	Face Map	Complex cracks
Z7	Music and Dance Diagram	Separate detachmen
Z8	Flying Sky Chart	Separate detachment
Z9	Ribbon chart	Substantial shedding
Z10	Book distribution map	Substantial shedding

The study selected a variety of disease types of murals, among crack diseases, murals with single cracks, murals with multiple cracks, and murals with complex cracks. In the peeling disease, two murals with single point peeling disease and two murals with multiple points and multiple parts peeling were selected for restoration analysis and analysis of system performance.

As shown in Fig. 9, the root-mean-square error of different models was compared with the peak signal-to-noise ratio. When the root-mean-square-error of the image was lower and the peak signal-to-noise ratio was higher, the image quality was better. The model’s root-mean-square error was less than 2.01 dB for the single crack mural damage macaque image and monk image. In comparison, the traditional SOM model had a root-mean-square-error of approximately 3 dB on average, while the more effective PConv algorithm had an error rate of about 2.78 dB. As for the peak signal-to-noise ratio comparison, the study model reached an average peak signal-to-noise ratio of 43.54 dB with increasing iterations, which was a maximum reduction of 44.5% in the root-mean-square-error compared with other algorithms. The peak signal-to-noise ratio was improved by 14.02%. With increasing iterations, the root-mean-square-error of the research model was 4.25 dB and 3.38 dB, respectively, while the TV algorithm was only 4.49 dB and 4.55 dB. The research model had a maximum reduction of 43% in root-mean-square-error compared with other algorithms. The peak signal-to-noise ratio was improved by 12%.

Fig. 9

Quantitative analysis and comparison of models.

Figure 10 shows the other two evaluation parameters in the quantitative analysis, which are feature similarity and spectral information similarity. The two evaluation parameters can quantitatively evaluate the naturalness of the image after restoration and the obviousness of the restoration marks to mural visual effect requirements. With the increasing complexity of image cracks and the increasing number of iterations, the FSIM of the study model averaged 0.997 dB. Compared with the other four algorithms, the FSIM value increased by a maximum of 1.12%, and the SR-SIM values increased by 0.8% and 0.61%, respectively. At the end of the iteration, the FSIM value of the studied model increased by 1.05% and the SR-SIM value increased by 0.54%. Combined with Fig. 9, the study’s improved SOM restoration algorithm had excellent restoration results for murals, and the restoration accuracy could be effectively improved by visualizing features for clustering and stratification and restoration.

Fig. 10

Comparison of feature similarity and spectral information similarity.

4.2 Empirical analysis of mural restoration systems

Before the empirical analysis for the mural restoration system, the performance analysis was first conducted for the mural detachment type disease restoration system. Additionally, the study analyzed the restoration effect through direct observation and qualitative analysis. The quantitative analysis utilized the TV algorithm and K-SVD algorithm, while the Criminsi algorithm was employed for comparative analysis of the restoration effect. The restoration index was selected as the peak signal-to-noise ratio evaluation index. The comparative evaluation index of the peak signal-to-noise ratio is shown in Fig. 11.

Fig. 11

Comparison of signal to noise ratio for shed disease repair.

In the comparison of the peak signal-to-noise ratio in Fig. 11, the research model achieved a peak signal-to-noise ratio of 43.62 dB during the restoration of the music and dance painting, which was 5 dB higher than the Criminsi model. The average peak signal-to-noise ratio of the research model was only 38.47 for six different mural types, which were all higher than the other four algorithms. The restoration times for different mural images are shown in Fig. 12.

Fig. 12

Repair time comparison.

Figure 12 shows the comparison of the repair efficiency evaluation of different algorithms with the studied algorithms. The slowest repair speed among the four algorithms was the TV model with an average repair speed of 148 seconds, the K-SVD model with an average repair speed of 78.38 seconds, and the traditional SOM model with an average repair speed of 82.67 seconds. The research model had the fastest average repair speed of 57.58 seconds, and the slowest repair speed of the image type was the color pattern map with only 113.77 seconds. The fastest repair speed of the image type was the human face map with only 17.81 seconds. Taken together, the improved algorithm of this study had less restoration time and higher efficiency, with about 40.42% improvement in efficiency assessment.

Figure 13 shows the annotation and repair effects of different algorithms when processing different samples. BI, EDSR and SRGAN were different in each sample. The overall SSIM of BI algorithm was about 0.7 dB. It showed that the strategy proposed in this study was effective in the comprehensive automatic annotation and repair of the cracks and falls of the mural paintings of ancient buildings.

Fig. 13

SSIM comparison.

The fresco restoration system was applied to a fresco restoration project and restoration model usage evaluation indexes were established, with comparative experimental evaluation in three aspects: usage effect, staff usage satisfaction, and post-repair visual effect score. The higher the score, the better the effect. The comparison results are shown in Table 2.

Table 2

Evaluation of the use effect of the repair system

/	/	Research model	Traditional repair methods	K-SVD repair system
First group	Use effect	92	84	90
	Staff satisfaction	96	82	89
	Visual effect score after repair	98	86	93
Second group	Use effect	93	87	92
	Staff satisfaction	98	81	87
	Visual effect score after repair	98	85	91

From the scores of different restoration methods through two groups of experimental subjects in Table 2, the restoration system improved by the study was better than the other two restoration methods in three aspects: effectiveness of use, staff satisfaction with use, and visual effect after restoration. In the overall evaluation index, the research model had an average score of 92.5 for the use effect, 97 for the system use satisfaction and 98 for the visual effect after repair. The score results indicated that the improved mural restoration system of this study was more effective and had a more accurate restoration effect on different mural detachment diseases when applied to actual use.

5 Conclusion

As a historically and culturally significant heritage protected by the Chinese government, the murals in ancient structures embody thousands of years of Chinese historical and cultural artifacts. Sadly, they are currently enduring varying degrees of damage. It is a trend to study the use of computer digital technology to assist historical relic scientists to protect the historical and cultural heritage of ancient buildings. Therefore, according to the characteristics of mural damage, this study established an automatic mural damage labeling system and repair system. The experimental results showed that the labeling system could correctly mark the range of different mural pathologies and carry out pathological restoration. Meanwhile, compared with other restoration algorithms, the root-mean-square error of the proposed model was reduced by 43.84% on average. The peak signal-to-noise ratio was improved by an average of 14.66%. The FSIM value and SR-SIM value of the research model increased by 1.05% and 0.54% in mural paintings with different degrees of crack lesions. The average time of the research model to repair mural lesions was only 57.58 seconds. The longest time to repair mural diseases was only 113.77 seconds, and the shortest time to repair mural diseases was only 17.81 seconds. The efficiency of the research model in mural restoration was very high, and the efficiency evaluation had improved by about 40.42%.

The research result’s significance lies in its provision of a comprehensive solution for automatically annotating and restoring ancient architectural murals. This solution achieves efficient, rapid, and high-quality cultural heritage protection. The advantages of this study are the remarkable repair effect, simple method and high efficiency. However, a limitation of the extraction process is that the range and parameters of the contour line determine the effectiveness of the extraction. Additionally, values and parameters must be frequently adjusted to accommodate differing restoration contents. A key research direction in the future is to determine the standard parameters.

In future research, it is advisable to further enhance the algorithm for better quality and effectiveness in restoration and explore its potential applications in other areas of conservation and restoration of cultural artifacts. In conclusion, the results of this study provide an effective method for the automatic marking and restoration of damaged murals in ancient buildings. After the application of actual restoration work and the establishment of evaluation indicators for system use, the average score of use effect was 92.5 points, the average score of system use satisfaction was 97 points, and the score of visual effect after restoration was more than 95 points.

References

Song

and Wang

, Artificial Intelligence-Assisted Fresco Restoration with Multiscale Line Drawing Generation, Complexity 2021(4) (2021), 1–12.

Magon

P.M.

and Lama

E.D.

, Material characterization, stratigraphy, textures, and painting techniques of the muralpainting The Allegory of the Industrial Development of São Paulo by Fulvio Pennacchi, Studies inConservation 64(7/8) (2019), 387–396.

Dinu

, Cortea

I.M.

, Ghervase

, Stancu

M.C.

, Mohanu

and Cristea

, Optoelectronic investigation of the mural paintings from Dragutesti wooden church, Arges County, Romania, Journal of Optoelectronics and Advanced Materials 22(5/6) (2020), 303–309.

Song

and Wu

, Research on virtual color restoration of complex building system based on discrete wavelet transform, Discrete Dynamics in Nature and Society 2022 (2022), 1–11.

Yin

, Sun

, Yu

, Su

and Dong

, Influence of particle size distribution of pigments on depth profiling of murals using laser-induced breakdown spectroscopy, Journal of Cultural Heritage 47(7-8) (2020), 109–116.

Tao

, Song

, Xu

, Zhang

, Wu

and Bai

, Back in time: digital restoration techniques for the millennium Dunhuang murals, Universities of Posts and Telecommunications 29(2) (2022), 13–23.

Purkait

, Ghorai

, Samanta

and Chanda

, A patch-based constrained inpainting for damaged mural images, Digital Hampi: Preserving Indian Cultural Heritage 4(1) (2018), 205–223.

Liu

, Li

and Wu

, Research on restoration algorithm of tomb murals based on sequential similarity detection, Scientific Programming (8) (2021), 1–8.

Tascon

, Mastrangelo

, Gallegos

and Marte

, Determination of materials and techniques involved in the mural paintings of San Miguel Church, Argentina, Journal of Raman Spectroscopy 48(10) (2017), 1356–1364.

10.

Madariaga

J.M.

, Costantini

and Castro

, CHAPTER 15: Analysis of the degradation of medieval mural paintings in the open air abandoned church of ribera, North of Spain (26) (2018), 213–233.

11.

Wang

J.L.

, Zhang

Y.D.

, Wu

Y.Q.

, Liu

H.L.

and Wang

X.D.

, Influence of salt/water ratio on emergence and developing rate of mural disruption, Surface Technology 46(2) (2017), 1–7.

12.

Tao

, Song

, Xu

, Zhang

, Wu

and Bai

, Back in time: digital restoration techniques for the millennium Dunhuang murals, The Journal of China Universities of Posts and Telecommunications 29(2) (2022), 3–23.

13.

Beddad

, Hachemi

, Postaire

J.G.

and Vaidyanathan

, An enhanced nonlinear filter and its applications to medical image restoration, International Journal of Biomedical Engineering and Technology 38(4) (2022), 345–360.

14.

Chen

, Ai

and Guo

, Inpainting algorithm for dunhuang mural based on improved curvature-driven diffusion model,, Journal of Computer-Aided Design & Computer Graphics 32(5) (2020), 787–796.

15.

, Intelligent restoration of ancient murals based on discrete differential algorithm, Journal of Computational Methods in Sciences and Engineering 21(3) (2021), 803–814.

16.

and Shui

P.L.

, Noise-robust color edge detection using anisotropic morphological directional derivative matrix, Signal Processing 165(11) (2019), 90–103.

17.

Adamik

, Goga

, Pavlovicova

, Babinec

and Sekaj

, Fast robotic pencil drawing based on image evolution by means of genetic algorithm, Robotics and Autonomous Systems (1) (2022), 148–159.