Information extraction from Visually Rich Documents using graph convolutional network

Abstract

Visually rich documents, such as forms, invoices, receipts, and ID cards, are ubiquitous in daily business and life. Various methods have been used to convey such diverse information, including text, layout, font size, or text position. Combining these elements in information extraction can improve the result performance. However, previous works have not effectively utilized the cooperation between these rich information sources. Text detection and recognition have been performed without semantic supervision (e.g., entity name annotation), and text information extraction has been performed using only serialized plain text, ignoring rich visual information. This paper presents a method for extracting information from such documents, which integrates textual, non-spatial, and spatial visual features. The method consists of two main steps and uses three deep neural networks. The first step, Text Reading, employs two CNN models (Lightweight DB and C-PREN) for OCR tasks, based on the state-of-the-art models DB and PREN, with two improvements. These improvements include reducing noise by removing the SE block of DB and integrating both context and position features in PREN. The second step, Text Information Extraction, uses a graph convolutional network, RGCN, for name entity recognition. Experiments on self-collected and two public datasets have demonstrated that our method improves the performance of the original models and outperforms other state-of-the-art methods.

Keywords

Graph Convolutional Network OCR Text detection text recognition NER

1 Introduction

Document understanding, especially for Visually Rich Documents (VRDs) has been attracted by more and more researchers, due to its important role in practical applications [15]. Documents, like invoices, receipts, business cards, or identification (ID) cards, are very common in day-to-day business workflows. They have massively increased both in number and complexity. Therefore, the automatic extraction of information from these document images is crucial to reduce manual effort and enhance the digitization process.

VRDs typically present information through textual and visual cues, e.g., text contents, layout, tabular structure, font size, and text positions [19]. Regarding the complexity of this structure, the visual information can be analyzed through spatial and non-spatial based features. The first features relate to geometry information like layouts or coordinates, while the second concerns text appearances, such as size or colors. Consequently, automatically extracting information from such documents is not a trivial task due to the difficulty not only in recognizing textual contents but also in determining relevant entity names (ID, code, seller, address, date, etc.).

Recently, there have been numerous studies focusing on this problem, which usually applied three phases, including Document Normalization, Text Reading, and Text Information Extraction (TIE) [45]. The first phase is necessary to normalize and segment the right region of interest (ROI) from the background. This allows for improving the performance of the next steps, especially for images captured in different situations (orientations, scaling …) and containing many other objects (even other VRDs) [27]. The second phase detects text regions, extract them from the document image, and recognizes the corresponding text. The results of this phase are affected by many factors, such as the diversity of scene text, the complexity of the background, and the interference factors [48]. The third phase extracts semantic information of recognized text, such as entities and relations, which is considered a specialized area in natural language processing (NLP). This problem can involve some NLP sub-tasks, for example, named entity recognition (NER), relation extraction (RE), event extraction (EE), salient facts extraction or rule-based methods [1]. In this study, we are interested in the last two phases: Text reading and informationextraction.

To tackle this problem, previous works frequently focus on the combination of non-spatial visual and textual features. A common solution is to apply Optical Character Recognition (OCR) at Text Reading to extract text contents, rule-based methods and/or NLP to identify entity names at Text Information Extraction. With the development of deep-learning technologies, many neural networks, like CNN and RNNs, have been applied for these steps, such as the work presented in [2 , 39] that provides efficient neural networks for OCR, or [6 , 45] for Name Entity Recognition(NER).

Although achieving promising results, these works lack the cooperation between text reading and information extraction. The former recognizes text without semantic supervision (e.g., entity name annotation), and the latter extracts information from only serialized plain text, not using rich visual information [45]. They merely transcribe the input [6], which overlooks important semantic information about spatial representation. Therefore, they are inadequate to capture all semantic facts of VRDs. For meaningful results, they should rely on the ability to incorporate all possible information, including textual, non-spatial, and spatial visual features. In this context, graph-based methods [9] are suitable solutions to deal with this complexity.

A graph is composed of nodes and connected edges, in which the first components delegate individual objects, while the second ones relate to relationships among those objects. Like many other data in the real world, we can model VRDs by this structure [19]. For example, we can use a graph of text segments to describe these documents, where each node (a segment instance) is characterized by textual (text contents), non-spatial (font and color), and spatial (text position) visual information; the edges are the distance among these nodes.

Graph neural networks (GNN) are an emerging framework in the deep learning community, which are designed to tackle this type of data. They have proven their efficiency in learning graphs for numerous areas, such as natural language processing, computer vision, drug discovery, and social networks [9]. This framework propagates information along edges to update nodes or edges using non-linear transformation and aggregation functions.

In this paper, we propose a method to read key information and NER of VRD images, which uses both their textual and visual features. We combined a CNN and a Represented—residual Graph Convolution (RGCN) to cover spatial and non-spatial visual features. CNN was used to extract text regions, contents, and their positions, while RGCN allows us to get NER of extracted key texts.

In summary, our contributions can be summarized as follows:

we proposed a method, named VRD-GCN, to extract key information from common structured document images, such as receipts, business cards, or ID cards.

we investigated a combination between Convolutional Neural and Graph Convolutional Networks to process all information and relations shown in VRDs, including textual, non-spatial, and spatial visual features.

we implemented the proposed methods (lightweight DB and custom PREN for Text reading, and RGCN for Name Entity Recognition), evaluated them on three datasets (VinText [26], MC-OCR [41], and a self-collected dataset), and highlighted the experimental results.

This paper is structured in the following way: In Section 2 we review related works. Section 3 describes our proposed method and related materials. We show and discuss our results in section 4. Section 5 is our conclusion and further work.

2 Related works

This section presents a review of recent literature on text reading and text information extraction from document images.

Text reading using OCR methods has become a very active research topic recently. According to Long et al. [20], the evolution of text detection undergoes three stages, including learning-based methods with multistep pipelines, methods inspired by object detection, and methods based on sub-text components [20]. The third category, which can give the highest accuracy, is based on an important idea: instead of detecting the whole text instance, we can detect sub-text components and assemble them later. For instance, Baek et al. [3] proposed a method supporting text detection, named Character Region Awareness for Text Detection (CRAFT), which relies on exploring each character region and the affinity between them. These character-level methods can be found in the work of Liao [18]. Component-level is another method that focuses on the local region of text instance [20]. For instance, Zhang et al. [46] proposed a Graph Convolutional Network to detect such text, which is based on the layout and geometry attributes of text.

For text recognition, since (scene) text images can be curved, deformed, or extremely long, thus before truly mapping them to corresponding strings, previous works usually performed several processing steps for more stable results. According to [8], the following steps are common for many SOTA text recognition methods:

Image preprocessing aims to enhance the image quality including background removal, text image super-resolution, and rectification. It allows for improving feature representation and recognition in the following steps.

Feature representation maps the text image (after preprocessing) to a representation reflecting the character attributes while suppressing irrelevant features such as font, color, size, and background. Convolutional neural networks and their variants are the most common method for this step, such as VGGNet, Recursive CNNs, ResNet, and so on.

Sequence modeling is an important step that combines the visual and contextual information of text images. It allows considering the character sequence as a whole, which makes the prediction more stable. Bidirectional long short-term memory (BiLSTM) networks are widely applied for sequence modeling.

Prediction, which includes two main techniques Connectionist temporal classification (CTC) and the attention mechanism (ATTN), attempts to map the target string based on the extracted features.

Text information extraction aims to extract useful information from recognized text, such as entities and relations. This problem can involve some NLP sub-tasks, for example, named entity recognition (NER), Relation extraction (RE), Event extraction (EE), and salient facts extraction [1]. They can be categorized into two main approaches: Rule-based methods (RBM) and learning-based methods (LBM) [1, 45].

RBM relies on (i) rules that are represented by a set of specific properties, and (ii) a corresponding interpreter that applies these rules to find suitable patterns [43]. The regular expression is the most popular applied technique to define rules, which contains a sequence of characters of expected patterns. For example, Profitlich et al. [32] recently proposed a case study that applied the NegEx regular expression algorithm to detect and extract negations of concepts in German medical texts like discharge summaries and clinical notes. Similarly, Giannakis et al. [13] presented a method to extract information from Linked Data, which applied w-regular expressions. The author proposed a novel way to query the potentially infinite graph of Linked Data. This technique can also be found in [7 , 42].

Regarding LBM, two main techniques have been applied in the literature: supervised, and unsupervised techniques. The need for labeled data is the most challenge for the first approach. Whereas, the second doesn’t need this data, but requires more intensive data processing [1]. Similarly to text reading, deep learning has been popular for TIE, which shows better results for studies with large datasets. Numerous applications in areas, such as name entity recognition, relation extraction, semantic analysis, searching, and classified models in natural language processing (NLP) have proven their effectiveness [35]. For instance, Nguyen et al. [25] applied transfer learning from transformers for Japanese NER in business documents. The authors successfully based on BERT [10] and ALBERT [16] to fine-tune transformers with a small amount of data. Zhang et al. [45] also presented an end-to-end method for IE, which contain a multimodal neural network (biLSTM) for TIE. The input is a combination of textual and visual features such as color, font, and layout of documents. Similar methods can be found in [5, 30].

Between RBM and LBM, the latter seems to have better performance in text information extraction. However, it still depends on the user needs and the task at hand because IE is a community-based process [1]. For example, with studies the dataset is not available or not enough, RBM seems a suitable solution.

Previous studies have shown the competency of the proposed methods on their tasks, most of them nevertheless considered text reading and text information extraction as two different phases that are executed independently. Although, several works tried to share and exploit the common features, such as visual and textual features. But, they still were not efficiently applied. Besides, information extraction from VRDs, which usually includes numerous models and interactive steps, requires a suitable architecture for their implementation in practice.

3 Material and methods

3.1 Overall methods

In this study, we are interested in extracting information from common Vietnamese VRDs, such as invoices, ID cards, or receipts. These VRDs describe their content using both textual and visual cues. Therefore, we propose a method, named VRD-GCN, that combines CNNs and GCN to recognize key texts and their semantics. The first networks were mainly used to detect, recognize, extract optical text, and supplement information, such as text coordination, width, or height. While the second network performs NER tasks.

The overall method is presented in Fig. 1. First, we use two networks, lightweight DB and custom PREN, to extract raw texts and their position. Second, these outputs are engineered to produce two sets of features: node and edge features, that incorporate textual, spatial, and non-spatial visual information. Lastly, we use these features to train the residual gate graph convolutional network (RGCN) to identify important keys. The next sections will detail these networks and features.

Fig. 1

VRD-GCN: a CNN and GCN based method for information extraction of VRDs.

3.2 Text reading: textual, non-spatial and spatial visual feature extraction

To incorporate all important information located on VRDs, we used two CNN-based neural networks to extract textual, non-spatial, and spatial visual features. First, we applied transfer learning with DB network [18] on two Vietnamese optical text datasets: VinText-2021 [26] and our self-collected dataset.

The detailed architecture of DB is presented in Fig. 2, which consists of three main components, including the backbone, neck, and head. The first component is used for feature extraction. To balance the accuracy and the computation time, instead of using ResNet as in the original network, we updated the backbone of this network to a lightweight one: EffecientNet B3 [38]. Moreover, we removed Squeeze and Excite (SE) block to reduce the noise due to its multiplication operation. After feature extraction, the second component processes these features to find potential areas that can include texts. The pyramid features are the main operation for this component, in which extracted features are processed by several up-sampling and 2D convolution blocks.

Fig. 2

DB network [18].

At the last component, the network predicts labels of processed features. To do that firstly, the network produces two maps, which predict (i) the probability of pixels being in text areas, and (ii) the threshold for the boundary of each area. Finally, the approximate binary map is calculated to predict the text areas. The output of this step contains both non-spatial (feature map of text areas) and spatial visual (coordination of bounding boxes) features.

For the textual features, we proposed a deep neural network, named C-PREN, to recognize extracted optical texts from the previous step. The proposed network was based on PREN [44], in which we imposed the idea implemented in ABINet network [12]. ABINet applied the principle of decoupling recognition models into the vision model (VM) and language model (LM) to analyze and combine visual and linguistic features. These sub-models were used as functional units independently and learned separately. In fact, the idea is not new and was applied previously [28]. But, ABINet proposed to integrate both the context and position information in LM, which allows for improving the performance of the overall model. Owing to the availability of such information, this method fits perfectly with VRDs.

The detailed architecture of C-PREN is presented in Fig. 3 with two inputs, instead of only text images as in the original network. They correspond to two types of information: text images and position (coordination of bounding boxes) for context and spatial information, respectively. First, text images are passed through the new backbone, EffecientNet B3 [38]. Then, C-PREN predicts the visual result using the PREN method, as illustrated in the upper branch in Fig. 3. We also down-sample extracted features from the PREN backbone into 1/32, 1/16, and 1/8 of size and send them to a Feature Pyramid Network. The obtained results are features of size 1/8 of the extracted features, that are then passed through a self-attention block [40].

Fig. 3

Custom PREN (C-PREN) and Feature fusion.

We combined these results and the positional information, then corrected them with a specific block, named Multi-Domain Character Distance Perception (MDCDP) [47]. This block allows for establishing a visual and semantic-related position encoding. The MDCDP was proposed in the CDistNet network [47], which is an improved version of the transformer encoder [40]. It supports focusing more attention on the location information, as presented in Fig. 4. Finally, we fed the two outputs of MDCDP and the original PREN into a fusion layer for customization and generating the text outputs, as illustrated in the last layers of Fig. 3. Embedded text and coordinates of the bounding box will be used to initiate an input graph. Subsequently, this graph will be passed to multilayer RGCN to construct a new graph of which node features are used to predict its label through a fully connected layer.

Fig. 4

MDCDP block in CDistNet [47].

3.3 Text information extraction: Residual gate Graph Convolutional Network for NER

To extract the entity names of recognized texts, we proposed a residual gate Graph Convolutional Network (RGCN) that allows to integration and process of spatial information and text contents from the previous steps. The input of this network is a graph whose nodes and edges are derived from extracted features and information from the previous steps, as present in Fig. 5. Figure 6 presents the way to extract node and edge features. From the predicted text and position, nodes, which are equivalent to a bounding box, have been initiated using their coordinates, length, and the corresponding embedded text. We applied an LSTM layer with 512 hidden units and tanh as its activation function to pack these features together. A node is considered to be connected to another if the distance between two center points along the y-axis is 3 times less than its width. Edges have been featured by calculating the distance between two center points of the bounding box along coordinate axes.

Fig. 5

Overall structure for NER.

Fig. 6

Node and edge feature engineering.

We then passed these features (nodes and edges) to n + 1 layers of RGCN networks. The number of layers depends on the complexity of structured documents: the greater the complexity, the higher the number of layers. For low-complexity documents like ID or business cards, 3 – 5 layers are reasonable, while for more complex documents, like receipts and invoices, 6 – 8 layers are a good choice. Of course, for low-complexity documents, it is still possible to use more layers, but it can lead to critical issues, such as vanishing gradients or reduced performance.

RGCN is a deep neural network that has common issues, including vanishing and exploding gradients. To overcome these issues, the output of each layer in RGCN was normalized and added to the next input, as illustrated in Fig. 7. Graph normalization block based on basic normalization. Each GCN inside the Residual gate GCN performed an iterator operation to construct a new graph, as presented in Fig. 8 and Equation 1.

Fig. 7

Structural of Residual gate GCN.

Fig. 8

Structural of Residual gate GCN.

$h_{i}^{k + 1} = h_{i}^{k} + \sum (h_{j}^{k} * n_{i, j}^{k})$ (1) Let n_(i,j) be the weight that presents the influence level between node i and node j, $h_{i}^{k}$ and $h_{j}^{k}$ be the feature of node i and j in the k layer, in which j is adjacent to i. The feature in the (k + 1) layer, $h_{i}^{k + 1}$ , was computed by the previous layers $h_{i}^{k}$ and its adjacency $h_{j}^{k}$ , as presented in Equation 1. The equation shows that the more layer of RGCN, the more nodes are affected by more distant nodes. This method is the same as the way humans perform, in which we can reach the important information on a structured document by key sentences/keywords (adjacent nodes). For example, in a Vietnamese receipt, the total cost is always a number that was put on the same line with the sentence containing the "total cost:". Thus, for this type of document, the total number will be highly impacted by the node that presents this keyword. $n_{i, j} = n_{i} * w_{i} + e_{i, j} * w_{i, j} + n_{j} * w_{j}$ (2) The influence level n_i,j in Equation 1 depends on features of the two nodes and the connected edge, as presented in Equation 2, where:

e_i,j is the features of the edge between two nodes i and j

n_i, n_j are the features of the node i and j

w_i and w_j are the adjacent matrix (or the weight matrix) of the node i and j.

To avoid the exploding gradient, the influence level of each adjacent node was passed to a sigmoid function. Basically, the label of a node can be recognized by information on adjacent nodes. But only a few of them keeps the keyword role, while the others help to define the relationship of that node to the whole document, such as the relative position of documents (top, bottom, left, or right). Therefore, the influence level of each adjacent node must be different depending on the contextual meaning which was presented in its features. By training on the real data, the influence level of each adjacent node was changed through the gradient descent method to match the considered structured document.

4 Experiments

To evaluate the performance of the proposed method, we performed experiments on three datasets, including the VinText dataset [26], a new synthetic dataset derived from our previous work [27, 28], and the MC-OCR dataset [41]. We evaluated the lightweight DB network on both the VinText and MC-OCR datasets, while C-PREN was assessed on all three datasets. For the NER task, we used only the MC-OCR dataset. The following sections will provide a detailed description of our experimental methodology.

4.1 Dataset

The VinText [26] is a dataset for Vietnamese text detection and recognition. The dataset contains 2,000 fully annotated images with 56,084 text instances. Each instance was described by a quadrilateral bounding box and its own underlying truth character string. The authors have randomly split the dataset into three subsets for training (1,200 images), validation (300 images), and testing (500 images). The dataset was either downloaded from the Internet or captured in everyday life in Vietnam. Therefore, it is very diverse, which consists of busy and chaotic scenes with many shop signs, billboards, and propaganda panels.

The synthetic dataset has been generated from a 40,000-word Vietnamese dictionary, of which 23,000 words contain 2 compound words and 17,000 single words. We have applied several image processing techniques to enrich the dataset, including changing angles, blur, contrast, brightness, and zooming of text images. Finally, we obtained 9 million images containing text. The dataset has been randomly split into three subsets for training (80%), validation (10%), and testing (10%).

Mobile captured receipts OCR (MC-OCR) dataset [41] is the dataset that was used in the RIVF2021 MC-OCR Competition, as illustrated in Fig. 9. It contains 2,436 Vietnamese receipt images. Each receipt has been manually annotated with two groups of ground truth: (i) the quality and important keywords. The quality of receipt images was determined by the line that annotators can easily extract, while four keywords were identified, including SELLER, ADDRESS, TIMESTAMP, and TOTAL COST. To improve the training performance, we adjusted a new keyword, named OTHERS, that describes the other recognized texts. The dataset has been split into three subsets for training (80%), validation (10%), and testing (10%) models.

Fig. 9

Vietnamese receipts.

4.2 Experiment setup

To evaluate the performance of the updated networks (the lightweight DB and C-PREN), as well as the overall method VRD-GCN, we conducted three scenarios of experiments, as follows:

Lightweight DB evaluation: We trained both the new and original networks using the VinText and MC-OCR dataset. Then, we compared the results obtained from the two networks.

C-PREN evaluation: This scenario is similar to the first scenario, but we trained the PREN and C-PREN using all three datasets.

VRD-GCN evaluation: VRD-GCN can be considered as an end-to-end process in which the OCR task is performed first to prepare the input for the subsequent step, Named Entity Recognition (NER). The final output of VRD-GCN includes five key entities (SELLER, ADDRESS, TIMESTAMP, TOTAL_COST, and OTHERS) and their recognized texts. To assess the effectiveness of the entire method, we rely on two measures: (i) the performance of NER, and (ii) the character error rates (CER) of the obtained texts. In this scenario, we used OCR results obtained from the MC-OCR dataset to evaluate RGCN, thus the NER performance. The predicted entity texts were used then to calculate CER.

The F1 score, precision, recall, and accuracy were used to evaluate the experimental results. CER is calculated as presented in Equation 3, where $L = \sum_{i = 1}^{N} l_{i}$ represents the total length of all reference texts in the test set, l_i is the length of the i^th document, and N is the total number of test samples. The Levenshtein distance is calculated as (i + s + d), where i, s, and d correspond to the minimal number of character insertions, substitutions, and deletions required to transform the reference text into the OCR output. $CER = \frac{1}{L} \sum_{i = 1}^{N} (i + s + d)$ (3) The models were implemented using the TensorFlow Framework 2.3.0 and Python 3.6.9 on an NVIDIA Tesla P100 PCI-16Gb and an Intel(R) 2.3Ghz Xeon(R) microprocessor. We determined the parameters for our experiments based on the original work of DB [18] and PREN [44], as well as similar studies such as [9 , 28]. This approach allowed us to build upon previous research and ensure that the experiments were conducted using established best practices in the field. The following parameters and techniques were used for training models:

The loss functions were the binary loss, loss-attention, and online hard example mining (OHEM) [36] for the lightweight DB, C-PREN, and RGCN, respectively;

The Adam optimizer was employed for optimization with β1 = 0.9 for all models; β2 = 0.999, and e = 10^-8 for the lightweight DB; β2 = 0.98, and e = 10^-9 for C-PREN; and β2 = 0.999, and e = 10^-7 for RGCN.

To minimize the cost function, we applied a mini-batch with a size of 8, 64, and 24 for the lightweight DB, C-PREN, and RGCN, respectively;

The maximal training epochs were set to 1,200, 1,500, and 1,000 for the lightweight DB, C-PREN, and RGCN, respectively.

Besides, shuffled data were used to allow all models to learn randomly and provide more objective results; the shuffle process was conducted before selecting the batches. The initiated learning rate was 10^-3, and a self-adjusting learning rate technique was applied for all optimizers.

4.3 Results and discussion

For the first and second scenarios, we conducted four experiments, whose results are provided in Table 1 (DB and lightweight DB comparison) and Table 2 (PREN and C-PREN comparison). From the obtained results, we found that the light-weight DB outperforms the original works by 3.61% in terms of the F1 score. This can be explained by the improvements that we have implemented in the proposed networks. The new backbone EfficientNet B3 applied in the lightweight DB allows us to obtain better features while removing the SE block helps the network avoid the effect of noise. We also compared the lightweight DB with OCR methods/tools that used the original DB. The obtained results show that our updated network outperforms these frameworks in text detection, as illustrated in Fig. 10. The figure shows an example of text detection using Paddle OCR and the lightweight DB. Our model can detect unclear or blurred texts while Paddle OCR ignores them.

Fig. 10

The F1 score and loss progress curves of RGCN model on MC-OCR dataset.

Table 1

The lightweight and original DB comparison

Model	Precision	Recall	F1 Score
Original DB	77.9%	71.2%	74.77%
Lightweight DB	78.56%	79.42%	78.38%

Table 2

C-PREN and PREN comparison

Model	Accuracy
Original PREN	83.6%
C-PREN	87.2%

C-PREN also outperforms the PREN applied on the same dataset by 3.6% in terms of accuracy, as shown in Table 2. Owing to the new backbone (EffecientNet B3) and especially the application of the Transformer technique [40], the new network is capable of combining visual and linguistic features. Therefore, the correction of predicted texts achieves a better performance.

For the third scenario, the results of NER are presented in Table 3. The table indicates that VRD-GCN achieves a high precision, recall, F1 score, and accuracy of 92.78% 93.19% 92.97% 92.93% , respectively. Among the four important classes, the SELLER, and TIMESTAMP labels achieve the best and worst performance with an F1 score of 95.25% and 86.55% respectively. This can be explained by the fact that the SELLER class is usually quite different from the others, while the format of TIMESTAMP is varied. Figure 11 shows the RGCN model training progress curves over time in terms of F1 score and loss. The training was stopped at the 49th epoch after the performance on the validation set had stopped increasing. The F1 scores are stable and achieve high values just after 5 - 10 epochs, while training and validation loss decrease as the number of training iterations increases. The gap between the loss curves is small, which indicates that no overfitting occurs.

Fig. 11

Text detection using Paddle OCR (the upper image) and the lightweight DB (the lower image).

Table 3

NER results on MC-OCR dataset

Class	Precision	Recall	F1 Score	Acc
SELLER	95.25%	95.25%	95.25%	95.47%
ADDRESS	92.44%	96.42%	94.39%	94.35%
TIMESTAMP	86.91%	86.19%	86.55%	86.02%
TOTAL_COST	90.6%	89.53%	90.0%	89.78%
OTHER	98.7%	98.56%	98.64%	99.04%
Average	92.78%	93.19%	92.97%	92.93%

VRD-GCN also achieves a low CER of 0.218 which indicates a high accuracy in recognizing the texts of the entities. This, combined with the high performance of the NER task, demonstrates the effectiveness of the overall method in accurately extracting the key entities and their corresponding content.

Table 4 compares VRD-GCN to the baseline method that used the original DB, PREN, and RGCN, as well as recent studies applied to the same dataset, MC-OCR. The improvements in the lightweight DB and C-PREN allow us to achieve better OCR results, compared to the baseline. This leads to a lower CER (0.218 compared to 0.275) and higher accuracy of NER (92.93% compared to 91.08%). Moreover, the proposed method also outperformed recent works in terms of either accuracy or CER. For example, Pham et al., who used CRAFT, TransformerOCR, and GNN-based methods, obtained a similar accuracy as ours, but with a higher CER. Overall, we achieved the lowest CER, while the accuracy of NER is superior among the reported values.

Table 4

Performance comparison

Methods/Tasks	NER	CER
VRD-GCN	92.93%	0.218
[22]	–	0.22
[23]	–	0.23
[31]	92.2%	0.25
[24]	–	0.26
Baseline method	91.08%	0.275
[17]	–	0.30
[4]	–	0.32
[14]	–	0.39

We also conducted experiments to evaluate the computation time of the proposed models on both GPU (Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz 1.80 GHz with Tesla P100 GPU) and CPU (Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz 1.80 GHz) configurations. The obtained results are presented in Table 5. The computation time of VRD-GCN is faster than the baseline method with a time of 29.07ms (30.43ms compared to 59.5ms) and 14.3s (11.4s compared to 25.7s) on GPU and CPU, respectively. On GPU, text detection takes the longest time (30 ms), while recognition takes 9s on CPU. Furthermore, the size of VRD-GCN is smaller than the baseline (141.5 MB compared to 417 MB), which allows us to deploy the obtained model in devices with limited storage.

Table 5

Computation time and model size comparison

Model	GPU	CPU	Size
DB	35ms	8.6s	110MB
PREN	22.7ms	16.7s	257MB
Lightweight DB	30ms	2s	80MB
C-PREN	2.75ms	9s	91MB
RGCN	1.8ms	0.4s	50MB
Baseline models	59.5ms	25.7s	417MB
VRD-GCN	30.43ms	11.4s	141.5MB

Despite achieving an overall high performance, VRD-GCN exhibits limitations in predicting TIMESTAMP and TOTAL_COST, resulting in lower F1 scores. These limitations can be attributed to the small size of the dataset (MC-OCR) and the variability in annotations for these classes. Consequently, the model may not have received sufficient training data to learn the relevant patterns, which leads to its lower performance compared to other classes.

5 Conclusion

In this paper, we have presented a method to extract information from VRDs, which is capable of using textual, non-spatial, and spatial visual features, usually available in these documents. The proposed method contains two main steps with three deep neural networks. The first step, Text Reading, relies on two CNN-based models to detect and recognize texts and supplement information, such as text coordination, width, or height. For text detection, we updated the DB network [18] with two improvements, including the new backbone EffecientNet B3 and removing the SE block. For text recognition, we propose the C-PREN network that was based on the work of [44]. In this new network, we implemented a new block, MDCDP, that allows the combination of visual and spatial features (text images and position information). The output of these models was then fused to build a graph describing studied VRDs, which contains node and edge features. In the second step, Text Information Extraction, we applied the RGCN network to process the output graph of the previous step and obtain important named entities. This network was based on Graph Convolutional Network. Consequently, the two steps can share and integrate the common features between them.

We conducted empirical experiments on three datasets for information extraction from Vietnamese receipts, including a self-collected and two public datasets. The obtained results indicate that our method achieved high accuracy and outperformed the current SOTA method. They show the promise of the proposed method in the information extraction of VRDs.

In future work, we plan to address the limitations of our study by incorporating additional datasets and improving our model architecture to better learn the complex patterns (i.e., TIMESTAMPS and TOTAL_COST). Additionally, we will focus on expanding the scope of our research to include other VRDs, such as medical documents, and improving the performance of our model, particularly in the areas of text detection and recognition.

References

Adnan

, Akbar

An analytical study of information extractionfrom unstructured and multidimensional big data, Journal of BigData 6 (2019)1–38.

Baek

, Kim

, Lee

, Park

, Han

, Yun

, Oh

S.J.

Lee,

What is wrong with scene text recognition model comparisons? dataset and model analysis (2019a). CoRR, abs/1904.01906. http://arxiv.org/abs/1904.01906.

Baek

, Lee

, Han

, Yun

, Lee

Charac-ter Region Awareness for Text Detection (2019b). CoRR, abs/1904.01941. http://arxiv.org/abs/1904.01941.

Bui

D.C.

, Truong

, Vo

N.D.

, Nguyen

MCOCR Challenge 2021: Deep Learning Approach for Vietnamese Receipts OCR. In 2021 RIVF International Conference on Computing and Communication Technologies (RIVF) (2021) (pp. 1–6). doi: 10.1109/RIVF51545.2021.9642128.

Carbonell

, Fornés

, Villegas

, Lladós

A Neu-ral Model for Text Localization, Transcription and Named Entity Recognition in Full Pages (2020).

Carbonell

, Riba

, Villegas

, Fornés

, Named

J.Lladós.

Named Entity Recognition and Relation Extrac-tion with Graph Neural Networks in Semi Structured Documents. In 2020 25th International Conference on Pattern Recognition (ICPR) (2021) (pp. 9622–9627). doi: 10.1109/ICPR48806.2021.9412669.

Catelli

, Gargiulo

, Casola

, De Pietro

Fujita

Esposito

Crosslingual named entity recognition for clinicalde-identification applied to a COVID-19 Italian data set, Applied Soft Computing 97 (2020)106779. https://doi.org/10.1016/j.asoc.2020.106779.

Chen

, Jin

, Zhu

, Luo

, Wang

Text Recognition in the Wild: A Survey (2020a).

Chen

, Tang

, Qi

, Li

C.-G.

, Xiao

Learning Graph Normalization for Graph Neural Networks (2020b). arXiv. doi: 10.48550/ARXIV.2009.11746.

10.

Devlin

, Chang

M.-W.

, Lee

, Toutanova

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019).

11.

, Li

, Guo

, Yin

, Liu

, Zhou

, Bai

, Yu

Yang,

Dang,

Wang,

PP-OCR: A Practical Ultra Lightweight OCRSystem (2020).

12.

Fang

, Xie

, Wang

, Mao

, Zhang

Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition. (2021) arXiv.doi: 10.48550/ARXIV.2103.06495.https://arxiv.org/abs/2103.06495.

13.

Giannakis

, Theocharopoulou

, Papalitsas

, Andronikos

, Vlamos

Associating -automata to path queries on Webs of LinkedData,115–123. Mining the Humanities: Technologies andApplications, Engineering Applications of Artificial Intelligence 51 (2016), 115–123. https://doi.org/10.1016/j.engappai.2016.01.013.

14.

Hieu

T.B.

, Duc Viet,

Hiep,

N.M.

Ngoc Bao Anh,

Bao,

H.G.

Duc Anh,

Bui,

H.-P.

Nguyen,

T.H.

Le Nguyen

Le,

T.-L.

MC-OCR Challenge 2021: A Multimodal Approach for Mobile-Captured Vietnamese Receipts Recognition. In 2021 RIVF International Conference on Computing and Communication Technologies (RIVF) (2021) (pp. 1–6). doi: 10.1109/RIVF51545.2021.9642088.

15.

Hwang

, Lee

, Yim

, Kim

, Seo

Cost-effective End-to-end Information Extraction for Semistructured Document Images (2021).

16.

Lan

, Chen

, Goodman

, Gimpel

, Sharma

, SoricutALBERT:

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2020).

17.

, To

, An

, Ho

, Nguyen

, Do

, Ngoand

T.D.

, Le

D.-D.

MC-OCR Challenge 2021: An end-to-end recognition framework forVietnamese Receipts. In 2021 RIVF International Conference on Computing and Communication Technologies (RIVF) (2021) (pp. 1–6). doi: 10.1109/RIVF51545.2021.9642121.

18.

Liao

, Wan

, Yao

, Chen

, Bai

Realtime Scene Text Detection with Differentiable Binarization (2019). CoRR, abs/1911.08947. http://arxiv.org/abs/1911.08947.

19.

Liu

, Gao

, Zhang

, Zhao

Zhao, Graph Convolution for Multimodal Information Extraction from Visually Rich Documents (2019). CoRR, abs/1903.11279. http://arxiv.org/abs/1903.11279.

20.

Long

, He

, Yao

Scene Text Detection and Recognition: TheDeep Learning Era 2020.

21.

Mande

, Yelavarti

K.C.

, JayaLakshmi

Regular Expression Rule-Based Algorithm for Multiple Documents Key Information Extraction. In 2018 International Conference on Smart Systems and Inventive Technology (ICSSIT) (2018) (pp. 262– 265). doi: 10.1109/ICSSIT.2018.8748764.

22.

Nguyen

C.M.

, Van Ngo

Nguyen

D.D.

MCOCR Challenge 2021: Simple approach for receipt information extraction and quality evaluation. In 2021 RIVF International Conference on Computing and Communication Technologies (RIVF) (2021a) (pp. 1–4). doi: 10.1109/RIVF51545.2021.9642150.

23.

Nguyen

D.-C.

, Nguyen

T.-A.

, Nguyen

X.-C.

MCOCR Challenge 2021: End-to-end system to extract key information from Vietnamese Receipts. In 2021 RIVF International Conference on Computing and Com-munication Technologies (RIVF) (2021b) (pp. 1–5). doi: 10.1109/RIVF51545.2021.9642083.

24.

Nguyen

H.V.

, Bao Doan

Trinh,

H.V.

Huy Phan

Thanh,

T.M.

MC-OCRChallenge 2021: Towards Document Understanding for Unconstrained Mobile-Captured Viet-namese Receipts. In 2021 RIVF International Conference on Computing and Communication Technologies (RIVF) (2021c) (pp. 1–5). doi: 10.1109/RIVF51545.2021.9642126.

25.

Nguyen

M.-T.

, Le

D.T.

, Le

Transformersbased informationextraction with limited data for domain-specific business documents, Engineering Applications of Artificial Intelligence 97(2021d), 104100. https://doi.org/10.1016/j.engappai.2020.104100.

26.

Nguyen

, Nguyen

, Tran

, Ngo

, Nguyen

Hoai,

Dictionary-guided Scene Text Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021e).

27.

Nguyen-Trong

An End-to-End Method to Extract Information fromVietnamese ID Card Images, International Journal of AdvancedComputer Science and Applications 13(3) (2022). doi: 10.14569/IJACSA.2022.0130371.

28.

Nguyen-Trong

, Tran

, Pham

, Tran-Anh

Nguyen-Thi-Tan

Improving Text Recognition by Combin-ing Visual and Linguistic Features of Text. In The 11th International Symposium on Information and Communica-tion Technology. SoICT 2022 (2022) (pp. 329–335). New York, NY, USA: Association for Computing Machinery. doi: 10.1145/3568562.3568624 New York.

29.

, Li

, Hung

P.C.K.

, Chang

StaResGRUCNN with CMedLMs: Astacked residual GRU-CNN with pre-trained biomedical language modelsfor predictive intelligence, Applied Soft Computing 113(2021), 107975. https://doi.org/10.1016/j.asoc.2021.107975.

30.

Oral

, Emekligil

, Arslan

, Eryiğit

Informationextraction from text intensive and visually rich banking documents, Information Processing Management 57(6) (2020)102361. https://doi.org/10.1016/j.ipm.2020.102361.

31.

Pham

V.D.

, Nguyen

L.Q.

, Pham

N.T.

, Nguyen

B.H.

, Minh Dang

D.N.

Nguyen

S.D.

Key Information Extrac-tion from Mobile-Captured Vietnamese Receipt Images using Graph Neural Networks Approach. In 2022 6th International Conference on Green Technology and Sus-tainable Development (GTSD) (2022) (pp. 232–237). doi: 10.1109/GTSD54989.2022.9989111.

32.

Profitlich

H.-J.

, Sonntag

A case study on pros and cons ofregular expression detection and dependency parsing for negationextraction from german medical documents, Technical Report 2021.

33.

Radford

, Kim

J.W.

, Hallacy

, Ramesh

, Goh

, Agarwal

Sastry

, Askell

, Mishkin

, Clark

Krueger

Sutskever

Learning Transferable Visual Models From Natural Language Supervision (2021).

34.

Shehzad

, Ul-Hasan

, Malik

M.I.

, Shafait

Named Entity Recognition in Semi Structured Documents Using Neural Tensor Networks. In DAS (2020).

35.

Shi

, Jianping

, Jie

Prospecting Information Extractionby Text Mining Based on Convolutional Neural Networks–A Case Studyof the Lala Copper Deposit, China,6, IEEE Access 6 (2018)52286–52297. doi: 10.1109/ACCESS.2018.2870203.

36.

Shrivastava

, Gupta

, Girshick

Training Region-based Object Detectors with Online Hard Example Mining (2016) arXiv. doi: 10.48550/ARXIV.1604.03540. https://arxiv.org/abs/1604.03540

37.

, Zhou

, Gai

Research and Implementation of Data Extraction Method Based on NLP. In 2020 IEEE 14th International Conference on Anticounterfeiting, Security, and Identification (ASID) (2020) (pp. 11–15). doi: 10.1109/ASID50160.2020.9271745.

38.

Tan

, Le

Q.V.

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks (2019). doi: 10.48550/ARXIV.1905.11946.https://arxiv.org/abs/1905.11946.

39.

Umer

, Mondal

, Pandey

H.M.

, Rout

R.K.

Deep features basedconvolutional neural network model for text and non-text regionsegmentation from document images, Applied Soft Computing 113 (2021), 107917. https://doi.org/10.1016/j.asoc.2021.107917.

40.

Vaswani

, Shazeer

, Parmar

, Uszkoreit

, Jones

Gomez,

A.N.

Kaiser

Polosukhin,

Attention is All You Need. In Proceedings of the 31st International Confer864 ence on Neural Information Processing Systems. NIPS’17 (2017) (pp. 6000–6010). Red Hook,NY, USA: Curran Associates Inc.

41.

X.-S.

, Bui

Q.-A.

, Nguyen

N.-V.

, Nguyen

T.-T.-H.

, Vu

MC-OCR Challenge: Mobile-Captured Image Document Recognition for Vietnamese Receipts. In Proceedings of the 15th IEEE-RIVF International Conference on Computing and Communication Technologies. RIVF ’21. IEEE (2021).

42.

Wang

W.M.

, Li

, Tian

Z.G.

, Wang

J.W.

, Cheng

M.N.

Extractingand summarizing affective features and responses from online productdescriptions and reviews: A Kansei text mining approach,149–, Engineering Applications of Artificial Intelligence 73(2018a), 162. https://doi.org/10.1016/j.engappai.2018.05.005.

43.

Wang

, Wang

, Rastegar-Mojarad

, Moon

, Shen

, Afzal

, Liu

, Zeng

, Mehrabi

Sohn

, Liu

Clinicalinformation extraction applications: A literature review, Journal of Biomedical Informatics 77 (2018b)49. https://doi.org/10.1016/j.jbi.2017.11.011.

44.

Yan

, Peng

, Xiao

, Yao

Primi-tive Representation Learning for Scene Text Recog-nition (2021). arXiv. doi: doi: 10.48550/ARXIV.2105.04286. https://arxiv.org/abs/2105.04286.

45.

Zhang

, Xu

, Cheng

, Pu

, Lu

, Qiao

, Niu

, WuTRIE:

End-to-End Text Reading and Information Extraction for Document Understanding (2020a).

46.

Zhang

S.-X.

, Zhu

, Hou

J.-B.

, Liu

, Yang

, Wang

Yin

X.-C.

Deep Relational Reasoning Graph Network for Arbitrary Shape Text Detection (2020b).

47.

Zheng

, Chen

, Fang

, Xie

, Jiang

Y.-G.

CDistNet: Perceiving Multi-Domain Character Distance for Robust Text Recognition (2021). arXiv. doi: 10.48550/ARXIV.2111.11011.https://arxiv.org/abs/2111.11011.

48.

Zhu

, Yao

, Bai

Scene text detection and recognition:recent advances and future trends, Front. Comput. Sci. 10(1) (2016)19–36. doi: 10.1007/s11704-015-4488-0.