A mixed image segmentation method based on intelligent equipment

Abstract

Taking Uyghur character recognition as an example, this paper use a method of image segmentation which combines traditional methods with CNN, and transforms and implements the Uyghur character on intelligent devices. Firstly, after analyzing the application of several general segmentation methods, this paper finds some shortcomings but also some ideas for Uighur segmentation. Then, starting from the characteristics of Uyghur language, such as structure, word formation and input habits, the author studies the idea of Uyghur adhesive language segmentation from the perspective of language characteristics, puts forward the basic algorithm of Uyghur symbol segmentation, and applies the Uyghur character adhesion segmentation based on minimum spanning tree and multi-queue primitive merging model to improve the segmentation efficiency. In addition, in order to solve the limitations of the traditional handwriting recognition framework of “preprocessing $+$ feature extraction $+$ classifier”, this paper puts forward a new solution of Uyghur handwriting recognition technology combining prior domain knowledge with CNN, constructs a Uyghur handwriting recognition model on the basis of CNN and random elastic deformation, and also improves the recognition rate of the system by combining domain knowledge with CNN. According to the experimental results, we can conclude that the mixed method proposed in this paper can effectively break the technical bottleneck of traditional methods, thus improving the efficiency of segmentation and recognition.

Keywords

Image segmentation method prior domain knowledge CNN character recognition deep learning

1. Introduction

Usually, the traditional handwriting recognition is based on handwriting basic database and segmentation and recognition technology, which mainly includes three parts: preprocessing, feature extraction and classification recognition. There are 2 methods for the traditional recognition process of Uyghur characters. One is to recognize the whole word directly, which has rich desirable features but needs to establish a large number of word databases; Another method is to segment words (or sequences of overlapping letters) first, and then recognize the single characters after segmentation. In recent ten years, researchers at home and abroad have got features from two directions: statistical recognition and structural recognition, and used artificial intelligence, neural network, Hidden Markov Model, HMM, Support Vector Machine, SVM, quadratic linear decision function, etc. As the core or a series-parallel classification method to classify and recognize, which has achieved high recognition results.

The algorithms for handwritten Uyghur language recognition are mainly based on the main body direction code features, stroke structure features, stroke relationships, direction line elements and other features. At present, most of the classification methods we use refer to Chinese characters using Euclidean distance classification, SVM, HMM, neural network and so on, and use a certain classification method or multiple classification methods in series and parallel for classification and recognition. Although these methods draw lessons from Chinese recognition methods, their recognition rates are limited both in theory and in practice due to the characteristics of language itself. At present, one of the key points in the research of Uyghur and other conglutinated characters recognition is to seek a multi-feature and multi-classifier fusion recognition method suitable for language features.

In fact, not only the traditional Uighur recognition method is limited by the characteristics of the characters themselves, but also the research of recognition technology has encountered bottleneck. Many languages that are widely used in handwriting recognition at present have also entered a bottleneck in improving the handwriting recognition rate. For example, after more than 40 years of unremitting efforts by researchers, HCCR has achieved great success. Taking the literature [1] as the example, the discriminant feature extraction method and the discriminant learning quadratic decision function classifier are used. On the challenging online and offline handwritten Chinese character data sets CASIA-OLHWDB and CASIA-HWDB, the best recognition rates of online handwritten single character recognition are 95.28% (DB1.0, 4037 Chinese characters), 94.85% (DB1.1, 3926 Chinese characters) and 95.31% (ICDAR2013 Competition DB), respectively 3755 Chinese characters), the best recognition rates of offline handwritten single character recognition are 94.20% (DB1.0), 92.08% (DB1.1) and 92.72% (ICDAR 2013 Competition DB). However, handwriting recognition, whether online or offline, still does not achieve high enough accuracy. The recent research shows that the recognition rate of the mainstream handwriting recognition software in the market has not reached high precision, the recognition rate of many well-known brands of handwriting input method software is less than 90%, the recognition rate of the best system is only about 95%, and the character set supported by many software systems is very incomplete. Under the text line/single word or overlapping/single word mixed writing recognition mode, the recognition rate of many systems drops sharply, and the recognition performance experience of related software products needs to be improved urgently. Therefore, we can draw the conclusion that handwritten Chinese character recognition is still an unsolved and challenging research topic [2].

With the rise of deep learning, the handwritten Chinese character recognition problem has brought great vitality and extremely effective solutions. For example, since 2011, the winners of two consecutive ICDAR (International Conference on Document Analysis and Recognition) handwritten Chinese character recognition competitions have all adopted methods based on deep learning or neural network. It is worth mentioning that in the ICDAR Handwritten Chinese Character Competition in 2013, Graham from The University of Warwick won the first place in online handwritten Chinese character recognition with the method of deep sparse convolutional neural network, and its recognition rate was greatly improved to 97.39%, while the team from Fujitsu Company won the first place in offline handwritten Chinese character recognition by using the improved CNN network, with a recognition rate as high as 94.77%. Whether online or offline HCCR, the results obtained by the above-mentioned methods based on deep learning are far ahead of the traditional methods, showing the great potential of deep learning in the field of character recognition. Compared with traditional methods, deep learning can achieve higher recognition rate, but it also has some problems, such as longer training and testing time, too large dictionary storage and so on [3]. Therefore, the application of neural network based on deep convolution in Uyghur text recognition is a very challenging research topic, and the research of its recognition technology needs to be further expanded and deepened.

2. Application analysis of several general segmentation methods

2.1 Recursive method of fast segmentation

Put forward by Okamoto et al., this method has achieved good results in the segmentation of printed English letters. The basic idea of this method is to first perform horizontal projection, segment vertically adjacent characters or character strings, and then recursively segment vertically projected characters, such as:

Hi Poly How are you

The segmentation process is as follows (Fig. 1).

Figure 1.

Recursive segmentation.

The greatest strength of this method is its simple algorithm and fast segmentation speed. However, if there is no obvious blank in the horizontal and vertical directions, this method is difficult to apply. In view of Uyghur handwriting input habits and other reasons, there are few cases that actually meet this requirement, so this method has only theoretical significance for Uyghur segmentation, and the actual segmentation efficiency is poor [5].

2.2 Window method

The basic idea of this method is to find a window suitable for stroke size by calculating the bounding box of strokes, and then use the window to segment symbols. The specific process is as follows:

(1)
The width and height of the bounding box of each stroke in the sample book.
(2)
Determine the size of a window, sum the width and height of the bounding box of all strokes and take the average value, and add a constant as the width and height of the window on the basis of the average value.
(3)
Use the window to judge whether the strokes belong to a character. If the strokes are in the same window, then they are judged as a group of strokes belonging to a character. This window is represented by the dotted line in Fig. 2 below.

Figure 2.
Window segmentation.

Through the analysis of window method, the accuracy of Uighur symbol segmentation is not high. In fact, the greater the difference in the size of the symbols to be recognized, the worse the segmentation effect will be, because the window method uses windows of uniform size to segment symbols. One of the important distinguishing features of Uyghur symbols and letters is the difference in aspect ratio, and in actual writing, one Uyghur letter often exists in the window of another letter, which easily leads to segmentation errors [6].

Martin, G once proposed a similar method of segmentation by sliding window, which is also a typical method based on recognition segmentation in handwriting recognition, and will not be described again.
2.3 Bounding box method

This method mainly takes into account that ordinary characters are mostly composed of more than one stroke, so believe that the bounding box of a group of strokes is more effective than the bounding box of strokes. As shown in Fig. 3, bounding box of strokes are used to determine stroke grouping.

Figure 3.

Bounding box of strokes

For example, the Uyghur symbol “Ø¦Ùˆ ” has two strokes, and two main points are specified for each stroke, which are marked with circles. They are the upper left point and the lower right point of the bounding box of each stroke, and the coordinates are $S_{1}(x_{\min 1},y_{\max 1})$ , $S_{2}(x_{\max 1},y_{\min 1})$ , $S_{3}(x_{\min 2},y_{max2})$ , $S_{4}(x_{\max 2},y_{\min 2})$ . For the whole symbol, the main coordinates are $S_{5}(x_{\min},y_{\max})$ , $S_{6}(x_{\max},y_{\min})$ , which are the upper left point and the lower right point of the bounding box of Uyghur symbols, as shown in Fig. 4 below.

Figure 4.

Example of bounding box of stroke group.

The attributes of bounding box are determined below, where Width represents the bounding box width and Height represents the bounding box height, so the calculation method of the bounding box range and attributes is as follows [7]:

$\displaystyle x_{\min}=\min(x_{\min 1},x_{\min 2})$ $\displaystyle y_{\min}=\min(y_{\min 1},y_{\min 2})$ $\displaystyle x_{\max}=\max(x_{\max 1},x_{\max 2})$ $\displaystyle y_{\max}=\max(y_{\max 1},y_{\max 2})$ $\displaystyle\textit{Width}=x_{\max}-x_{\min}$ $\displaystyle\textit{Height}=y_{\max}-y_{\min}$

Using the bounding box information of characters, the segmentation process mainly depends on two characteristics, that is, the position relationship of strokes and the time between two strokes. Among them, the positional relationship of strokes can be divided into two types, namely, cross relationship and relative relationship. In case of there are one or more intersections for two strokes (including one bounding box contained in the other), then the two strokes are cross-related, and they are judged to belong to the same group of strokes; Otherwise, we can judge them by relative relation. As shown in Fig. 5 below.

Figure 5.

Cross-relationship.

Figure 6.

Three cases of relative relationship.

There are 3 cases for the relative relationship: The first case is that there is no intersection between two strokes, but the distance between them is within a given threshold range, so the two strokes also belong to a group of strokes. The first case is that two horizontal lines with similar sizes exist in two strokes. The third case is one of the two strokes is one or several points, and the other stroke is above (or below), then they belong to the same stroke group. As shown in Fig. 6 below.

Compared with the window method, the bounding box method has been improved, mainly because it does not use a uniform window size for segmentation, but uses the stroke bounding box intersection method, so this method is more efficient than the window method in character size difference when segmenting characters. In Uyghur handwritten symbols, some symbols meet the conditions of this method, but there are also quite a few symbols which are intersected by too many bounding boxes, so this method alone can not meet the requirements, but this method provide an effective solution [8].

2.4 Dynamic programming algorithm

The basic process of segmentation using dynamic programming is to find a path in the graph and separate each character in the graph. This path of character segmentation is the segmentation point to be found, that is to say, the segmentation process is to search for the best path. Dynamic programming algorithm is one of the most effective methods in solving the best path at present. Dynamic programming is a multi-step decision-making process. This method simplifies the N-step process into N single steps, which greatly reduces the computational complexity. Dynamic programming method needs to determine its search space, cost function and search method [9].

2.5 Method based on soft decision

This method constructs a virtual network, and calculates the probability that a stroke group belongs to or constitutes a character on the basis of defining the type and frequency of each stroke.

From the above analysis, it can be seen that all these segmentation methods have certain advantages, especially for characters with similar size and relatively independent symbols, but they also have some problems. Combining the characteristics of Uighur symbols, this paper puts forward a technical method to improve the segmentation performance [9].

3. Segmentation method based on CNN and multi-queue primitives

3.1 Application of multi-queue primitive merging model

In this paper, combined with primitive processing and based on contour features, a multi-queue primitive merging model for Uyghur symbol segmentation is proposed, so as to improve the segmentation accuracy.

The basic steps of over-segmentation based on contour features are as follows:

For a Uyghur handwritten sequence, the object of over-segmentation is the main stroke (that is, the symbol main body within a certain threshold range of the baseline). Then the main stroke is obtained and its connected domain is segmented to make it a primitive. Generally, additional strokes may not be segmented, because additional strokes are already primitives in general. Over-segmentation is a method of segmentation using the related features of contour, which can find all suitable segmentation points with high anti-interference. The main purpose of finding contour features is to find the feature points of the contour, and then apply structural element rules to filter the segmentation points. The structural elements of Uyghur symbols mainly include holes, additional symbols, symbols above the baseline, symbols below the baseline, symbols across the baseline, turning points, etc. The process of searching for segmentation points in Uyghur is as follows:

(1)
Selecting the local lowest point of the upper contour as a candidate segmentation point.
(2)
The point where the straight line distance from the upper contour to the lower contour is within a certain threshold range is selected as the candidate segmentation point.
(3)
Similar to Rule 2, all points whose distance from the lower contour to the upper contour is within a certain threshold range are selected as segmentation points.
(4)
The intersection of the upper contour through which the baseline passes is selected as the segmentation point, and Hough transform method is used for baseline detection.
(5)
Filter the segmentation points obtained above, and the final segmentation points should ensure that the straight line distance between any two candidate points should be less than a threshold, otherwise, delete one of the candidate segmentation points.

These segmentation rules are based on the characteristics of Uyghur symbols, that is, adjacent letters in a Uyghur word are usually connected with each other on the baseline (within a certain range). Based on this phenomenon, the above rules select the points on the baseline as the initial segmentation points. Theoretically, these three different characteristics can find almost all segmentation points, which include local minima found according to rule 1, bottleneck points of upper and lower contours found according to rule 2 and rule 3, and baseline intersection points found according to rule 4. Although the candidate segmentation points searched by using these rules as a whole will have redundancy, its advantage is to minimize the missing points [10].

Considering the difficulty of Uyghur primitive merging, we use the multi-queue primitive merging model to deal with the complex Uyghur primitive merging problem. The processing of this model mainly includes stroke subject merging, stroke addition merging and stroke addition and stroke subject merging. In the whole merging process, geometric information is combined, and comprehensive information of recognition and logic is applied to analyze the rationality of the final merging scheme.

For the queue model of Uyghur symbols, we divide primitives into three parallel queues, namely, the main body, the queue above the baseline and the queue below the baseline, thus forming a three-dimensional state space. All segmentation schemes of primitives are described by state path. Thus the merging of Uyghur primitives becomes a mathematical problem of path optimization, and the symbolic confidence is calculated by various feature information. In this paper, the dynamic programming method is used to optimize the path segmentation.

The basic idea is to simplify the complex layout of primitives into multiple parallel primitive queues, and then describe and optimize the primitive merging scheme [11, 12].

Set the sequence ${e}_{i}(0<i\leqslant N)$ as the primitive queue of the main part, the $e_{i}^{u}(0<i\leqslant N^{u})$ as as the primitive queue on the baseline and $e_{i}^{d}(0<i\leqslant N^{d})$ as the primitive queue under the baseline. It is assumed that any legal Uyghur symbol can be merged by the primitives of the above three sequences, and that the primitives in the same sequence are all adjacent. That means any Uighur symbol sequence ${c}_{k}(0<k\leqslant N)$ corresponds to a segmentation path $x_{k}=({x_{k},x_{k}^{u},x_{k}^{d}})$ , so that

$\displaystyle c_{k+1}=\{{e_{i}|x_{k}<i\leqslant x_{k+1}}\}\cup\{{e_{k}^{u}|x_{% k}^{u}<i\leqslant x_{k+1}^{u}}\}\cup\{{e_{i}^{d}|x_{k}^{d}<i\leqslant x_{k+1}^% {d}}\}=c_{k+1}\cup c_{k+1}^{u}\cup c_{k+1}^{d},0\leqslant k<N$

Among them,

$\displaystyle 0=x_{0}\leqslant x_{1}\leqslant\cdots\leqslant x_{N}=N$ $\displaystyle 0=x_{0}^{u}\leqslant x_{1}^{u}\leqslant\cdots\leqslant x_{N}^{u}% =N^{u}$ $\displaystyle 0=x_{0}^{d}\leqslant x_{1}^{d}\leqslant\cdots\leqslant x_{N}^{d}% =N^{d}$

For vector sets $x_{k}({k=0,1,\ldots,N})$ , satisfy

$\displaystyle x_{k+1}=x_{k}+u_{k}({x_{k}\in X,u_{k}\in U,0\leqslant k<N})$

Among them,

$\displaystyle{X}=\{{({x,x^{u},x^{d}})|0\leqslant x\leqslant N,0\leqslant x^{u}% \leqslant N^{u},0\leqslant x^{d}\leqslant N^{d}}\}$

${u}_{k}$ and ${x}_{k}$ are both three-dimensional vectors, but the difference is that the value range of ${u}_{k}$ should be limited in the set $U$ , where

$\displaystyle{U}=\{{({u,u^{u},u^{d}})|u\geqslant 1,u^{u}\geqslant 0,u^{d}% \geqslant 0}\}\cup\{{({0,0,0})}\}$

There are restrictions on the first and last states of the split path $\{{x_{k}}\}$ , including the first state $x_{0}=({0,0,0})$ and the last state $x_{N}=({N,N^{u},N^{d}})$ .

The purpose of segmentation is to optimize the objective function:

$\displaystyle\textit{conf}=\frac{\sum_{i}n_{i}\cdot\textit{conf}_{i}}{\sum_{i}% n_{i}}$

This is the confidence weighted average of independent symbols. In which $n_{i}$ is the weight of the corresponding symbol $c_{i}$ and the number of primitives that constitute the symbol $c_{i}$ . $\textit{conf}_{i}$ is the confidence of the symbol $c_{i}$ [13].

Then there is the process of using dynamic programming to solve the best path. First, the dynamic programming of one dimensional element merging is solved as follows:

In this primitive merging model, any legal Uyghur symbol is formed by merging a continuous group of primitives in the primitive sequence. It is assumed that there are N pieces of Uyghur symbol primitives $e_{i}({0<i\leqslant N})$ , which have been ordered from left to right, then any Uyghur symbol sequence $c_{k}({0<k\leqslant N})$ assembled from this primitive queue will correspond to a unique segmentation sequence $x_{k}$ ( $k=0,1,\ldots N$ ), that is:

$\displaystyle c_{k+1}=\{{x_{k}<i\leqslant x_{k+1},0\leqslant k<N}\}$

Among them,

$\displaystyle 0=x_{0}\leqslant x_{1}\cdots\leqslant x_{N}=N$

According to the primitive model of Uyghur segmentation, the one-dimensional dynamic programming is extended to the three-dimensional dynamic programming of Uyghur symbol primitives, namely:

For the 3D Uyghur symbol primitive segmentation sequence $x_{k}(k=0,1,\ldots N)$ , here is its dynamic expression:

$\displaystyle x_{k+1}=x_{k}+u_{k}({x_{k}\in S,u_{k}\in U,0\leqslant k<N})$

In which $k$ represents discrete time and $x_{k}$ represents the state in $S$ (three-dimensional segmentation state space), where,

$\displaystyle{s}=\{{({x,x^{u},x^{d}})|{0\leqslant x^{u}\leqslant N^{u},}0% \leqslant x^{d}\leqslant N^{d}}\}$

$u_{k}$ represents the control decision vector at $k$ , which is limited in space $U$ .

$\displaystyle U=\{({u,u^{u},u^{d}})|{u\geqslant 1,u^{u}\geqslant 0,u^{d}% \geqslant 0}\}\cup\{{0,0,0}\}$

The initial state and the final state of this dynamic system are limited to $x0=(0,0,0)$ and ${x}_{N}=(N,N^{u},N^{d})$ , respectively. Define the confidence measure of $cf(c)$ for symbol $c$ , and make it satisfy $cf(c)\leqslant 0$ . If the confidence of a symbol sequence is the weighted sum of the confidence of each symbol, and the weight is the sum of the primitive numbers of each symbol, that is, $cf(c)|c|$ ( $|c|$ is the primitive number in set $c$ ). Then the objective function at this time can be further expressed as:

$\displaystyle g_{N}\left({x_{N}+\sum_{k=0}^{N-1}g({x_{k},u_{k}})}\right)$

Among them,

$\displaystyle{g}({x,u})=\left\{{{\begin{array}[]{ll}cf({.c|_{x,u}})\cdot||u_{1% }||_{1},&u\in U\text{ and }x\in S\\ -\infty,&u\notin U\text{ or }x\notin S\\ \end{array}}}\right.$ $\displaystyle g_{N}(x)=\left\{{{\begin{array}[]{ll}0,&x=({N,N^{u},N^{d}})\\ -\infty,&x\neq({N,N^{u},N^{d}})\\ \end{array}}}\right.$

And $||u||=|u|+|u^{u}|+|u^{d}|$ .

The process of optimization is to find the maximum objective function $k_{k}^{\ast}({k=0,1,\cdots N})$ , and then use the improved dynamic programming algorithm to solve the problem globally.
3.2 CNN

3.2.1 Uyghur handwriting recognition model based on CNN and random elastic deformation

CNN is composed of input layer, hidden layer and output layer. And the hidden layer is divided into three categories: 1. Convolution layer for feature extraction; 2. Sampling laye for feature optimization and selection; 3. Hidden layer in traditional multilayer perceptron. In accordance with the characteristics of Uyghur, especially the problem of too many similar characters, the input images are convolved with 8 trained filters (K $\times$ K convolution kernel), and 8 images (28 $\times$ 28 in size) are obtained, each image representing the features obtained by filtering with 8 different filters. The input of convolution layer is connected to the neurons of the previous layer through a K $\times$ K convolution kernel (as connection weight). Each convolution layer is composed of multiple feature maps, and each feature map can be connected with multiple feature maps of the previous layer. Convolution kernels between feature maps are shared by neural units in the same feature map and can be determined by learning. Neurons in the same feature map share a bias.

The sampling layer improves the anti-noise ability by reducing the resolution of the feature map, that is, by sampling the feature map of the convolution layer to extract important classification features, while ignoring irrelevant details. At present, the sampling layer of CNN model mainly adopts two construction methods: subsampling (using the sampling method of average value) and maximum pool (using the sampling method of maximum value). Each neuron in the sampling layer is connected to the neighboring neurons in the 2 $\times$ 2 range of convolution layer. Through convolution layer and sampling layer, the dimension of feature map will gradually decrease, but the number will gradually increase. The random diagonal LM algorithm is proposed for CNN learning. Generally, this two-step descent learning algorithm can make the training of the model converge quickly, but when the number of learning iterations is less than 20, the error function can converge to a satisfactory result. The random diagonal LM algorithm adopts different learning rates for each connection weight, and the learning rates are adjusted according to the second derivative of the parameters to be learned, that is,

$\displaystyle\eta_{ki}=\frac{\varepsilon}{\frac{\partial^{2}E}{\partial w_{ki}% ^{2}}+\mu}$

Where $\varepsilon$ is the global learning rate, which decreases with the number of learning times. $\mu$ is used to avoid the problem that the denominator is too small and the learning rate is too high. Specific values will be determined based on the experimental test results in the future [13, 14, 15].

3.2.2 Uyghur handwriting recognition method combining domain knowledge with CNN

In order to avoid over-fitting in the training process and improve the recognition performance of CNN model, in addition to some classical methods such as Dropout, obtaining sufficient training samples is the prerequisite to ensure the high performance of CNN model. However, the training samples obtained are very limited in many cases. Therefore, we should consider applying data addition technology to improve the robustness and popularization ability of CNN system. Based on affine transformation, this paper further discuss the elastic deformation data generation technology applied to online Uighur recognition. Theoretically, these methods can enrich the number of samples in the sample space without changing the internal structure of the original character sequence, and control the degree of sample transformation to make the generated samples conform to the actual sample distribution.

Based on the sparse CNN model, this paper further introduces the path integral feature to act on the input layer of CNN as an on-line feature extraction method of time series information. In theory, the introduction of path integral feature map can make the recognition result on the data set obviously superior to that of MCDNN. In the design, it is considered to further add relevant domain knowledge such as nonlinear normalization, virtual stroke technology, eight-direction feature and path integral feature map as prior knowledge. For example, virtual stroke technology is mainly used to weigh the weight distribution between real strokes and virtual strokes, while eight-direction feature and path integral feature are mainly used to complement each other to improve recognition performance.

4. System prototype and implementation technology based on intelligent equipment

The related techniques of Uyghur symbol segmentation are proposed in this paper, and the implementation techniques of the system mainly include MFC application and Windows Mobile (WM) development technology. This paper will introduce the prototype system and its development and running environment, and explain the Visual Studio IDE using MFC class library. In addition, in order to apply some related algorithms based on windows platform to smart devices, this paper also studies and analyzes the event-driven and message response mechanism of WM, as well as the key steps of porting Windows programs to WM programs, and summarizes and analyzes many differences between WM programs and Windows programs.

4.1 Visual Studio IDE

Visual Studio can create applications and network services under Windows environment, in addition it can create applications, network services and Office plug-ins for smart mobile devices. Besides, Visual C++ can not only be integrated with Visual Studio, but Visual C++ can be installed separately for convenience.

Visual Studio contains a variety of enhancements, such as language enhancements that can handle almost all data types, visual designers, and a large number of improvements in Web tools. Visual Studio packages a wealth of tools and frameworks for developers to use, so as to facilitate the creation of applications.

Visual Studio integrates MFC. MFC class library encapsulates some Windows API in the form of C language classes, and also contains a complete program framework to help developers reduce their workload. MFC includes many encapsulation classes of Windows handles and encapsulation classes of Windows built-in components.

4.2 Event and message mechanism of WM

WM programming also belongs to event-driven design mode. WM program provides users with many visual objects. By selecting any possible functions, the functions selected by users will generate corresponding events. Then, the event transmits the message to the specific object of the program, and then the object performs the corresponding function by calling the message function. The visual interface provided by WM application is also regarded as an object, and the operation performed on the visual object will trigger the corresponding function through an event. The process in which WM application runs is the process in which external users operate to form events, and the events are processed accordingly.

The event mechanism of WM originates from Windows. Loading message and response message are the unique mechanisms of Windows core. In Windows applications, messages respond to events, and events form messages. This processing mechanism is between users and systems, between systems and programs, and between programs, as shown in Fig. 7 below.

Figure 7.

Message response mechanism.

4.3 Differences between WM program and Windows program

It has been proved that most Windows applications can be ported to WM, and the workload of porting is much less than rewriting these programs, especially those based on MFC. Compared with Windows applications, there are many differences in developing WM applications. based on the Uighur handwriting recognition system, the differences between WM and Windows programs are analyzed as follows [16, 17, 18]:

4.3.1 The difference between Microsoft Win32 API and Windows CE API

Win32 API includes Windows CE API, that is, Windows CE API is a subset of it, and many of its functions have been simplified. For example, in the programming of Uyghur handwriting recognition system, it is necessary to make a file on Windows CE API that correctly displays and splices Uyghur fonts on mobile devices, because its support for fonts is limited.

In addition, some extended functions of Windows CE API, such as touch screen and notification, need hardware support.

There are limitations in exception handling of Windows CE. Although it supports structured handling mechanism, it does not support exception handling of C++.

When porting Win32 programs to the target Windows CE platform, the most time-consuming task is that there is no corresponding Windows CE API to replace Win32 API, and the related functions need to be redesigned.

4.3.2 The difference between Microsoft MFC and Windows CE MFC standards

Although Windows CE MFC is in line with the specification of Microsoft MFC, it is quite different in the structure and function of providing classes. In addition, some features in Windows CE MFC are only for Windows CE. In this paper, to transplant Windows applications, it is necessary to systematically check the classes and their methods, attributes and scopes used by the programs and determine their compatibility with Windows CE MFC.

4.3.3 The scope and critical value of variables

WM mobile devices have small memory, CPU is slower than PC, and the scope and critical value of variables are different to some extent, so special attention is required in programming.

4.3.4 Memory limitation

Generally, the memory of mobile devices is smaller in capacity and speed than PC. In most cases, it is necessary to reduce the size of the application to migrate to WM. In this paper, when relevant codes are transplanted to WM, necessary functions such as sample data collection, recognition of Uyghur script mother and conjoined segments are modified, while relatively complex functions such as segmentation are no longer considered.

4.3.5 Treatment of testing and debugging process

The programming work is carried out in WM simulation environment, which is different from the practical application in intelligent devices. Generally, before an application is officially used, it must be tested on all the devices that are expected to run the application, not just relying on the simulation environment.

4.3.6 Compact Framework tools

Compact Framework simplifies the development process of intelligent terminal applications, and provides virtual devices such as Pocket PC, WM, Pocket PC Phone and other devices in windows CE.NET environment.

The NET Compact Framework procedure is created on Visual Studio .NET in this paper, and a Windows procedure is created on Visual C++ .NET.

4.4 Key factors for porting windows programs to WM

We don’t annotate the specific steps of transplantation in detail here, but will clarify the key points, which need to be handled according to the requirements of the program in practice.

4.4.1 Find the corresponding Windows CE API

Although Win32 can usually backwards compatibility Win16 functions, the simplified Windows CE can’t. Check API references in programs (such as functions, messages and data types, etc.), and replace or modify references that are not supported by Windows CE API. For example:

Win32 functions is not supported. If there is a replacement function, it will be updated as a replacement function. If not, it is necessary to establish a function to replace its functions, such as LineTo function and MoveTo function, which are no longer supported by Windows CE, and the function PolyLine is used to replace them.

The same function function of Windows CE replaces the original function function of Win32, for example, CommandBar function.

Win32 functions with some limitations. For example, the parameters of some functions have changed or the scope of the parameters has changed, such as CreateWindow function and CreateWindowEx function, whose window style has changed.

Some data types need to be modified.

Many messages are no longer supported. For example, many WM_* messages and EM_* messages are no longer supported, some changes have taken place in wParam, and some new messages have been added to Windows CE, such as WM_HIBERNATE.

4.4.2 Windows CE memory

For Windows CE applications, we should minimize the use of memory, simplify or even eliminate functions or files that consume large amounts of memory, and try not to use or keep temporary files as much as possible. Even sometimes, you can rewrite the code to reduce the speed in exchange for reducing the use of memory, and find a suitable compromise.

If there aren’t enough system resources, the WM_HIBERNATE message of Windows CE will inform the corresponding program. If the program receives this message, it will release resources and return the memory to normal level as much as possible. The key of this process is WM_HIBERNATE. A robust program should handle WM_HIBERNATE well and respond by releasing temporary files when resources are scarce.

4.4.3 Power consumption management

Mobile devices mostly run on batteries, so try to avoid unnecessary calls to CPU by code in programs. The use of PeekMessage function needs to be cautious, which will make CPU cycle run almost continuously.

4.4.4 Graphical user interface

Many Win32 GDI functions are no longer supported in Windows CE, so they need to be changed before transplantation.

4.4.5 Adjust bitmap and icon

Windows CE devices have smaller touch screens and different sizes, so it is necessary to modify the program to adapt to similar restrictions. It should be noted that the static layout should not be arranged as much as possible, but the screen size should be defined after obtaining the size through the function GetSystemMetrics.

4.4.6 Unicode encoding

Due to Windows CE itself is a Unicode coding environment, and it also supports calling ASCII functions, it can facilitate the exchange of text and files, which also facilitates the test of the algorithm in this paper. The conversion method from ASCII program to Unicode program is as follows:

Include Tchar.h, so as to include all necessary transformations.

Choose Win32 string processing function, such as lstrlen function, instead of using c runtime library.

Clearly select TCHAR, LPTSTR and other types in the statement. Make the code easier to compile into Unicode and ASCII.

select TEXT and _T macro to map the character string, such as text (“Uygur Pen”).

Make sure that the length of characters is not 1 byte, and the character string ends with 2 zeros.

To ensure the validity of Unicode and ASCII, sizeof(TCHAR) should be used when adding array pointers and counting characters.

4.4.7 Management window

Windows CE has few window styles and functions, so there is no available resize processing, and the window can only be the size defined when it was created.

5. Experimental results and analysis

5.1 Experimental evaluation index

This paper continues to use the following four groups of indicators to characterize the performance of the system (there are two calculation modes I and II), so as to evaluate the advantages and disadvantages of the algorithm. See Reference [19] for detailed indicator system and data set.

(1)
Accuracy CRSR
(2)
Error Segmentation/Recognition Rate ERSR
(3)
Segmentation/Recognition Rejection Rate RRSR
(4)
Average Time of Segmentation/Recognition ATSR
(5)
Segmentation/Recognition Accuracy PSR

5.2 Experimental conclusion and analysis

5.2.1 Segmentation results of common syllables

In this experiment, we made a statistical analysis of the segmentation and recognition operations of the two test sets. Among the 400 commonly used syllables of random data, 366 were correctly segmented, and 20 syllables were segmented. However, there were also cases of incorrect segmentation or partial segmentation errors, and 14 syllables were rejected. Therefore, according to the second mode,

The segmentation accuracy is:

$\displaystyle\textit{CRS}_{\textit{RCD}}=\frac{\textit{NCS}_{\textit{RCD}}}{% \textit{ANS}_{\textit{RCD}}}\times 100\%=\frac{366}{386}\times 100\%=94.82\%$

Segmentation error rate is calculated according to mode II:

$\displaystyle\textit{ERS}_{\textit{RCD}}=\frac{\textit{NES}_{\textit{RCD}}}{% \textit{ANS}_{\textit{RCD}}}\times 100\%=\frac{20}{386}\times 100\%=5.18\%$

$\displaystyle\textit{ERS}_{\textit{RCD}}=100\%-\textit{CRS}_{\textit{RCD}}=(10% 0-94.82)\%=5.18\%$

The segmentation rejection rate of available RCD test data is:

$\displaystyle\textit{RRS}_{\textit{RCD}}=\frac{\textit{NRS}_{\textit{RCD}}}{% \textit{TNS}_{\textit{RCD}}}\times 100\%=\frac{14}{400}\times 100\%=3.50\%$

Similarly, four groups of index values of SCD test data can be obtained, as shown in Table 1.

Table 1
Index value of segmentation test

Datasets/indicators	CRS (%)	ERS (%)	RRS (%)	ATS (ms)
RCD	94.82	5.18	3.50	/
SCD	98.75	1.41	1.73	/

5.2.2 Single character recognition results

The statistical results of RCD and SCD identification experiments are shown in the following table.

Table 2

Test results of RCD and SCD

Datasets/indicators	Correct number	Correct rate	Number of errors	Error rate	Rejection number	Rejection rate	Total
RCD	2011	89.94%	207	9.26%	18	0.81%	2236
SCD	491	94.79%	21	4.05%	6	1.16%	518

Based on the above two tables, compared with the previous traditional methods, we can conclude that CNN’s segmentation ability and recognition ability have been improved to a certain extent, and the rejection rate has been significantly reduced. (The result of the traditional method is that the segmentation error and rejection ratio is close to 14%, and the recognition rate of single character is less than 90%, see Reference [19] for details.) Moreover, this paper has completed a relatively complete test on smart devices for the first time. Although the data set has not changed, it also verifies the effectiveness of the hybrid method proposed in this paper on smart devices to a certain extent. The major problems encountered in this paper are that the time and space complexity of the algorithm is relatively high, and the battery power consumption is large, and the equipment heat generation is large. Next, we plan to continue to optimize the algorithm to reduce the time and space complexity of the algorithm, and test it on the latest basic database Ucpen2.0.

Footnotes

Acknowledgments

This work was supported by Natural Science Foundation of Xinjiang Province (No. 2018D01A28), Scientific Research Program of the Higher Education Institution of XinJiang (XJEDU2017M021), National Natural Science Foundation of China (No. U1903215, 61363062), Xinjiang University Research Foundation(No. BS180250).

References

Njah

Bezine

and Alimi

A.M.

, On-line arabic handwriting segmentation via perceptual codes: application to MAYASTROUN database, in: International Multi-Conference on Systems, Signals and Devices, IEEE, 2011, pp. 1–5.

Mustafa

M.E.

and Alshafy

H.A.A.

, Characters’ boundaries based segmentation for online Arabic handwriting, in: International Conference on Computing, Electrical and Electronics Engineering, IEEE, 2013, pp. 306–310.

Ibrayim

and Hamdulla

, On-line handwritten uyghur word recognition using segmentation-based techniques, International Journal of Signal Processing Image Processing & Pattern Recognition 8(6) (2015), 51–60.

Ibrayim

Hamdulla

and Tursun

, A dynamic programming method for segmentation of online cursive uyghur handwritten words into basic recognizable units, Journal of Software 8(10) (2013), 2535–2540.

Wujiahemaiti

Mayire

and Askar

, A survey on the classifiers in on-line handwritten Uyghur character recognition system, International Journal of Hybrid Information Technology 9(3) (2016), 189–198.

Wujiahemaiti

Mayire

and Askar

, Survey on the features for recognition of on-line handwritten uyghur characters, International Journal of Signal Processing, Image Processing and Pattern Recognition 8(9) (2015), 45–58.

Abdeen

R.M.

Afifi

and El-Sisi

A.B.

, An Enhanced Technique for Offline Arabic Handwritten Words Segmentation, in: Computational Linguistics and Intelligent Text Processing, Springer International Publishing, 2015, pp. 663–681.

Yin

Wang

Q.F.

et al., An over-segmentation method for single-touching Chinese handwriting with learning-based filtering, International Journal on Document Analysis & Recognition 17(1) (2014), 91–104.

Boubaker

Chaabouni

Halima

M.B.

et al., Arabic diacritics detection and fuzzy representation for segmented handwriting graphemes modeling, in: Soft Computing and Pattern Recognition, IEEE, 2015, pp. 71–76.

10.

Nakkach

Hichri

Haboubi

et al., A Segmentation-Free Approach to Strokes Extraction from Online Isolated Arabic Handwritten Character, in: International Conference on Advanced Technologies for Signal&Image Processing Atsip, 2016.

11.

Maqqor

Halli

Satori

et al., Using HMM Toolkit (HTK) for recognition of arabic manuscripts characters, in: International Conference on Multimedia Computing and Systems, 2014, pp. 475–479.

12.

Khemiri

Kacem

and Belaid

, Towards Arabic Handwritten Word Recognition via Probabilistic Graphical Models, in: International Conference on Frontiers in Handwriting Recognition, IEEE, 2014, pp. 678–683.

13.

Yong

and Pan

, A method for segmenting uyghur symbols, International Journal of Performability Engineering 13(6) (2017), 985–997.

14.

Elzobi

Al-Hamadi

Al Aghbari

and Dings

, IESK-ArDB: a database for handwritten Arabic and an optimized topological segmentation approach, International Journal on Document Analysis and Recognition, 2013, 295–308.

15.

Saabni

R.M.

and El-Sana

J.A.

, Comprehensive synthetic Arabic database for on-off-line script recognition research, International Journal on Document Analysis and Recognition 16 (2013), 285–294.

16.

Mahmoud

S.A.

Ahmad

Al-Khatib

W.G.

Alshayeb

Parvez

M.T.

Märgner

and Fink

G.A.

, KHATT: an open arabic offline handwritten text database, Pattern Recognition 47 (2014), 1096–1112.

17.

Omer

M.A.H.

and Ma

S.L.

, Recognition Online Arabic Pattern, in: Proc. Conf. on Advanced Computer Theory and Engineering, 2010, pp. 18–22.

18.

Assabie

and Biqun

, Offline handwritten amharic word recognition, Pattern Recognition Letters 32(8) (2012), 1089–1099.

19.

Yue

Pan

and Yong

, A handwritten uygur character string recognition method combining segmentation and whole word recognition, Technical Journal of the Faculty of Engineering 39(10) (2016), 86–98.