Automatic domain terminology extraction and its evaluation for domain knowledge graph construction

Abstract

Domain terminology recognition and extraction is the primary work for the construction of domain knowledge graph. Traditional method is tedious, and time-consuming, as well as low accuracy. This paper presents an improved Domain Term Extraction-Improvement (TDE-I) method based on relative co-occurrence rate, which can automatically extract domain basic terminologies and domain compound terminologies. We also present a Document Classification Value (DCV) method on the basis of calculating Domain Feature Vector (DFV) value which can implement a judgment to evaluate extraction accuracy according to the evaluation indexes. Our experimental results demonstrate that our approach is effectiveness and accuracy. The proposed method can provide with a new solution for the construction of domain knowledge graph and its applications.

Keywords

Knowledge graph domain terminology extraction extraction algorithm evaluation

1. Introduction

Nowadays witnesses a burst growth in quantities of on-line data and knowledge with rapidly developing of the Internet, knowledge expression and organization are being confronted with new challenges. In the year of 2012, Google launched its knowledge graph programme to establish a semantic knowledge base system [10,18], which brought a new vitality to semantic search applications. Knowledge graph [1] is therefore well-recognized to be a promising technical way to deal with these problems and obtains a great continued concern in the community of industry and academics [4,7,12,16,17].

Knowledge graph is broadly divided into two categories, i.e. general knowledge graph and domain knowledge graph. Different from general knowledge graph, coverage of domain knowledge graph is relatively narrow, limited to a particular industry area, but with a strong professional emphasizing on relationship between knowledge and knowledge. This paper focuses on the study of domain knowledge graph. Actually, domain terminology recognition and extraction is the primary step and plays a very important role in the construction of knowledge graph. The traditional methods mainly extract knowledge through a wide range of resources of Internet. In general, taking advantage of large quantity of volunteers in Internet, and using some tools provided, like freebase, Wikidata etc., domain terminologies are entered and knowledge relationships are established by manual, or knowledge is recognized and extracted based on some industry-specific data sources (such as professional website, domain-related internal database etc.) with the help of some professional tools such as Protégé etc., but also in a manual way. Although the results are accurate and quality is relatively high by manual means, drawbacks are very obvious actually. It is required large quantity of man-power and time, and also it is difficult to ensure that there will be no human errors. Under the constant updating and growth of knowledge, manual way obviously cannot be timely updated. More importantly, it would be even more difficult to establish a domain knowledge graph on the situation of absence of relevant professional talents. Thus, automatic extraction of domain terminologies through data mining massive data from Internet is essential towards the success of knowledge graph construction and deployment.

In order to solve the problems of discovery and extraction of domain terminology in establishment of domain knowledge graph, many scholars and scientists proposed their approaches and solutions. The research work on domain recognition and extraction was initiated by Luhn in 1958 [9]. Until now, the solutions can be primarily grouped into three categories, i.e. statistics based method [2,8,11], language rule based method [14,15] and semantic rule based method [3,20], as well as semi-automatic rule based method in the recent [13]. However, these existing methods generally have some issues on the aspects of low accuracy and large invalid words in domain terminology extraction. In order to solve these problems, this paper presents a novel solution to automatic discovery and extraction of domain terminology. On the basis of satisfying with accuracy in a certain threshold, those words that appear in the extracted results with high frequency but not real domain terms are removed, and then domain compound terminologies are constructed using the extracted domain basic domain terminology, to implement discovery and extraction of domain terminology automatically. Comparing with the previous extraction algorithms by some experiments, our method is proved to save a lot of manpower costs and time costs, improve accuracy and remove number of invalid words, which can provide with a better data source for the next stage of construction of knowledge graph.

The rest of the paper is organized as follows. Related work is reviewed in Section 2. Section 3 presents our approach to domain terminology extraction. Section 4 presents our evaluation method. Some experiment results are given in Section 5. Conclusions are drawn and future work is discussed in Section 6.

2. Related work

The most basic and the primary solution is statistics-based method [11]. Sentences are parsed into several words by a sentence splitter, and noise and trivial words, such as auxiliaries, adjectives, modal words, and so forth, are cleaned. Then, occurrence frequency of each meaningful word is counted and sorted, and those words that are located within a specified threshold range are treated to be domain terminologies. This solution is easy to perform, but having low accuracy, because terminology set inevitably contains a large amount of high frequency of writing words that have low relevance with domain. To solve this problem, some researchers began to find the improved methods, for example, Fukushige and Noguchi [2] proposed TF-IDF based weight computation method, taking both word frequency and document frequency into consideration. In spite of increased accuracy comparing to the simple statistic-based method, the extracted set still contained high rate of low domain relevance words. Lopes et al. [8] proposed an advanced method called TF-DCF to evaluate the relevance of terms extracted from domain corpora based on the absolute term frequency and disjoint corpora frequency. This is an efficient way to extract terms which are relevant for a specific domain, but it is based on the assumption that a term relevance to a specific domain can only be established by comparison with corpora from other domains (contrasting corpora).

Language rule-based method takes use of lexical rules explicitly. From the viewpoint of linguistic semantics, all general types of word combination models are limit which can be enumerated. For example, a sentence always contains a syntactic structure consisting of multiply nouns or gerund followed by verbs, thus words can be split out from sentences, marked, and consequently recombined to construct possible substrings which is recognized as domain terminologies. Following the principle of language combination, Shinnou et al. [15] made a progress through summarizing domain terminology syntactic rules. Saedi et al. [14] enrich source and target multiword terms with syntactic structure and seamlessly integrate them in the tree-based transfer phase of TectoMT. However, simple linguistic combination method can also not avoid invalid words essentially, leading to lower accuracy as the same as statistics-based method. Moreover, different domain has different language rule, and summarization of regularity of all domains will also be a tedious job.

Semantic structure can be useful. Homayouni et al. [3] proposed Latent Semantic Indexing (LSI), a popular linear algebraic indexing method to take advantage of implicit higher-order structure in the association of terms with documents (semantic structure) in order to improve the detection of relevant documents. This method can reach a higher accuracy rate than simple TF-IDF method. Subsequently, researchers are currently developing semantic rule-based method which combines semantic analysis with machine learning, and is helpful for a computer program to understand meaning of texts written in natural language, and to extract domain terminology automatically. This method is promising, but, still remains at the beginning stage which is being encountered some great difficulties.

3. Domain terminology extraction

3.1. Invalid terms elimination

The selection of corpus is the fundamental step from which domain terminology extraction can be initiated, and documents in corpus directly determine the accuracy of extraction. In this paper, some guideline rules are considered as follows for the selection of documents. 1) The document should be composed of a combination of sentences with consecutive contextual relationships; 2) The documents which are selected into corpus should be related to one domain theme, and have obvious domain differentiation, such as portal under various categories of articles; 3) The document should have a suitable length, being a combination of multiple consecutive sentences with contextual relationships, such as a complete news, papers, microblogging and so on. For a Chinese document, the suitable length will be more than 100 but less than 1,000 Chinese characters; 4) The document providing for the domain knowledge should be accurate and credible; and 5) The contents provided by corpus should have a large varieties.

Following these guideline rules, we select documents as samples in corpus from complete news, essays and blogs or published journals which are related to the focused target domain.

Definition 1 (Invalid term).

In daily life and writing, there are a large number of frequently used words, which having no actually meaningful, or having not too much impacts on expression after the removal of main meaning. These words are called invalid words. Such words appear in relatively high frequency, but they will be doped in the final results and affect accuracy of results. Therefore, invalid words must be handled for denoising before texts are processed. Three kind of words have to be filtered, like,

Punctuation mark;

No practical meaning word, such as auxiliary verb, modal word;

Non-word, such as adjective, distinguishing word, status word, numeral word, quantifier, adverb, preposition, conjunction, auxiliary, interjection, modal word prefix, and suffix, with no significant impacts on the meaning of expression, mainly playing a modified role;

Word with length of 1, considering that the possibility for a word, whose length in word segmentation being 1, becoming a domain terminology is very small.

After eliminating invalid terms, these corpus texts already have had the characteristics as follows. Each record is independent. There is no strong correlation between record and record. Also, each record contains the result of clause of the article that represents the record in form of array. Each clause also contains the result of word segmentation of the sentence in form of array. In construction of corpus, we use BeautifulSoup to parse html content to get pure plain contents, and also NLPIR lib [21] to split each sentence in documents into words. The results compose our corpus finally. The records of each document in the corpus are stored in Mongodb, NoSQL database with json format. Therefore, we can organize structure of each record in document as shown in Fig. 1, including id, title, content and frequency of word.

Fig. 1.

The structure of document of json in Mongodb.

3.2. Domain basic terminology extraction

Definition 2 (Domain basic terminology).

Domain basic terminology describes a vocabulary in a field of science and technology which has been widely accepted.

Traditional extraction algorithm combines co-occurrence rate and TF-IDF value to obtain basic terminology.

TF-IDF is the most classical algorithm which is widely used for domain terminology extraction. It is a statistical method, which is used to evaluate importance degree of a word in a single file or in all files of corpus. This method is composed of term frequency (TF) and inverse document frequency (IDF), being actually TF * IDF, the higher the occurrence frequency in a separate file is, the more important the word is, that is, the greater the weight is. In corpus, on the contrary, the higher the frequency is, the smaller the weight is. TF-IDF is actually a kind of weighted technology in field of information retrieval. If a word appears in some articles with high frequency, but frequency of appearance in corpus is not high, then the word would be probably the keyword of the article. It is required to assign a higher weight to distinguish it. The main issue of this algorithm is that, it is more suitable for it to extract keywords in article and remove invalid noise words and make classification on article according to extracted keywords, but for the strong domain corpus, because domain terminology in the whole corpus appears in a relatively high frequency, resulting in a lower TF-IDF value of domain terminology. Although TF-IDF is a simple and feasible algorithm, accuracy is not very high, which is mainly used for denoising and weight calculation for candidate terminology.

Therefore, we make improvements on the algorithm as follows and propose TDE-I (Term Domain Extraction-Improvement) algorithm. On hand, add seed terminology into the extraction of domain basic terminology to improve domain correlation of the results. On the other hand, instead of the existing co-occurrence rate with relative co-occurrence rate, adjacent position of co-occurrence word is also involved in the weighted adjustment. After obtaining domain terminology, it is participated in iteration calculation again as a seed terminology. Words can be ordered by their TF-IDF values. Some words having the highest TF-IDF value, calculating by Eq. (1), can be used to describe domain feature, where $w_{i}$ is the total number of words in the provided corpus. $\begin{matrix} (1) & TF - IDF = (w_{1}, w_{2}, \dots, w_{n}) \end{matrix}$

Definition 3 (Seed terminology).

Our TDE-I algorithm starts from several domain basic terminologies, to search for more related domain terminologies. These initial domain basic terminologies are called seed terminology.

Extracted terminology is often closely related to the words of seed terminology. The fit degree as well as relationship tightness between seed terminology and domain directly effects on degree of correlation between terminology extraction and domain. Thence, seed terminology plays a very important role in extracting domain terminology. The closer the relationship between the given seed terminology and the domain is, the more accurate the result is.

Definition 4 (Relative co-occurrence rate).

Relative co-occurrence rate is defined as ratio of number of co-occurrences and number of occurrences of co-occurrence words in corpus, calculating by Eq. (2). $\begin{matrix} (2) & S (x) = \frac{f (x, y)}{w (x)} \end{matrix}$ where $f (x, y)$ represents number of occurrences that the word appears in the text of all short sentences at the same time. $w (x)$ is number of times a word appears in all texts. For the same sentence, if x appears multiple times, it is still counted as 1.

The co-occurrence rate in the traditional algorithms tends to add a large number of common and ineffective words to the results, such as the terms of as, of etc. Different from co-occurrence rate, relative co-occurrence rate is able to remove some noise words that appear in the results with high frequency and invalid words.

Starting from seed terminology, we search for the words with a relatively high co-occurrence rate and eliminate the words under a certain threshold. And then we can calculate the TF-IDF value for these words in corpus and sort them, and eliminate some of them under a specific threshold. After two rounds of natural selection, the two invalid words mentioned before are moved, which can be effective to solve the problem of removing invalid words highly correlated to seed terminology in both traditional co-occurrence rate algorithm and TF-IDF algorithm. Through paragraph labeling technology, the weight can be adjusted based on adding position of the word and seed terminology. At the same time, those words often appearing one after the other at the same time are recorded. When frequency appearing simultaneously one after the other exceeds a threshold, they are directly combined to construct a new domain terminology.

Definition 5 (Candidate terminology).

Starting from seed terminology, the first results obtained by the algorithm are called candidate terminologies. As these terminologies are not conducted by weight calculation, part of speech filtering and threshold filtering and other operations, therefore, there are still a large number of invalid words and with low accuracy in candidate terminologies, it is required to confirm as a final domain terminology after filtering.

Suppose set of seed terminology of domain terminology is B, and set of candidate terminology is W, our algorithm is described as follows.

Step 1: Remove any arbitrary word x marked as unused from B and mark the status of the word as used. Traversal all words, find the word y whose relative co-occurrence rate is bigger than $f {(x, y)}_{min}$ and frequency of occurrence is bigger than ${TF}_{min}$ of x, and also and does not belong to the collection W. Store it in the collection W.

Step 2: Remove any arbitrary word k marked as unused from W, and mark it as used. Traversal word segmentation by sentence as the basic unit, and record the distance between k and x and also the TF-IDF value of the word. Save these data. When number of times that distance between k and x is 0 exceeds a certain threshold, and there is no combination of $k + x$ in B, then it is regarded that the word and x can be combined into a new domain terminology. Store the terminology into W.

Step 3: If there are unmarked words in W, then go to Step 2; otherwise go to Step 4.

Step 4: If there are still unmarked words in B, then go to Step 1; otherwise go to Step 5.

Step 5: Calculate the weighted frequency of each word in W by Eq. (3). $\begin{matrix} (3) & F (y) = \frac{f (x, y)}{w (x)} \times \frac{TFIDF (y)}{\sum_{i = 0} dtc (x, y)} \end{matrix}$ where $\frac{f (x, y)}{w (x)}$ is the relative co-occurrence rate between word y and word x. $TFIDF (y)$ is the TF-IDF value of word y in the whole corpus. $dtc (x, y)$ is the distance between word y and word x. In the paper, it is called domain correlation of the word y in domain. The greater the value is, the greater the relevance of the word y to the domain is.

Step 6: Sort the results after the weighted frequency in descending order. Make the word within a threshold range as a new seed terminology, and append it into the set W.

Step 7: Go to Step 1. If going through the Step 6, there is no new seed terminology in W and B, then the algorithm will be ended.

The set W obtained by the above algorithm is the set of domain basic terminology.

3.3. Domain compound terminology extraction

Definition 6 (Domain compound terminology).

Domain compound terminology is defined as a collection of multiple domain terminologies, which is a combination of domain basic terminology and other words. A whole vocabulary usually occurs frequently in an article. Based on it, starting from the acquired domain basic terminologies, we can traverse corpus. At the location that each domain basic terminology appears, combine it with its words before and after and regard it as a domain compound terminology. Filtering some terminologies that obviously cannot be a combination of compound terminology by the combination of part of speech. And then adjust the weight through statistics and iteration to obtain domain compound terminology.

Set domain basic terminology set as W and initial domain compound terminology set as M. Suppose $W M$ be a collection of candidate domain compound terminology. The steps of extracting domain compound terminology is described as follows.

Step 1: Remove any arbitrary domain basic terminology X and mark it as unused from W, traverse the result of word segmentation by sentence as the basic unit. When X appears in the word segmentation, set the previous position of the word which appears in the word segmentation is $X_{pre}$ , and the next position of the word is $X_{next}$ .

Step 2: If X is the first word of the word segmentation, then perform from Step 3 to Step 5. If X is the last word of the word segmentation, then perform from Step 4 to Step 5; otherwise conduct Step 3 to Step 5.

Step 3: Let the combination of the word X and the word $X_{next}$ be the word $X_{X + next}$ , and part of speech is the combination of $X . flag$ and $X_{next} . flag$ . If there have been already $X_{X + next}$ records in M, then the $X_{X + next}$ record add 1; otherwise create the $X_{X + next}$ record and set the number of occurrence as 1.

Step 4: Let the combination of the word X and the word $X_{pre}$ be the word $X_{pre + X}$ , and part of speech is the combination of $X_{pre} . flag$ and $X . flag$ . If there have been already $X_{pre + X}$ records in M, then the $X_{pre + X}$ record add 1; otherwise create the $X_{pre + X}$ record and set the number of occurrence as 1. The implementation step is basically the same as Step 3.

Step 5: If there are still unlabeled domain terminology in W, then go to Step 1; otherwise go to Step 6.

Step 6: Remove any unused domain compound terminology K from $W M$ , mark it as used. Check its compound part of speech. If it does not belong to any of $n n$ , $n v$ and $v n$ , then remove K from $W M$ .

Step 7: If there are still unmarked domain compound terminology in $W M$ , then repeat Step 6; otherwise go to Step 8.

Step 8: Sort domain compound terminology in $W M$ by descending order according to number of occurrences. Take a compound word set S within a certain threshold, where S is the resulting domain compound terminology set.

After several iterations of the above steps, the result will be a domain compound terminology which is a combination of the two words. Iterative results may still contain some incomplete domain compound terminologies. As incomplete domain terminology usually includes relationship with a complete domain terminology in corpus, it should combine a complete domain terminology with the words matching the incomplete domain compound terminology and its word before and after. In order to get a complete domain compound terminology, taking the iteration of domain compound terminology as a set of B, repeat the algorithm. After multiple iterations, we can get a complete domain compound terminology.

4. Evaluation method

4.1. Evaluation index

Currently, there is not yet a uniform standard for evaluation of terminology extraction algorithm. In the paper, basis of assessment we use is accuracy rate, and unrecognized rate. Statistics of extracted terminology is shown as Table 1.

Table 1
Statistics of extracted terminology

Type Terminology Non-terminology

Extracted a b

Un-extracted c

Type	Terminology	Non-terminology
Extracted	a	b
Un-extracted	c

The above situations are based on the standard glossary. Suppose standard glossary be $ST$ , and extracted terminology set be T, then $\begin{matrix} (4) & a = |ST \cap T| \end{matrix}$

Definition 7 (Accuracy rate).

Accuracy rate AR is defined as a degree of accuracy, calculating by Eq. (5). $\begin{matrix} (5) & AR = \frac{a}{a + b} = \frac{|ST \cap T|}{|T|} \end{matrix}$

Definition 8 (Unrecognized rate).

Unrecognized rate UR is used to obtain the comprehensive degree of extraction, calculating by Eq. (6). $\begin{matrix} (6) & AR = \frac{a}{a + c} = \frac{|ST \cap T|}{|ST|} \end{matrix}$

4.2. Domain Feature Vector (DFV)

After corpus selection and data preprocessing, we can have a corpus of a document collection D, an individual document $d_{j} \in D$ , remained word in $d_{j}$ document $W_{d j}$ and an individual word $w_{i, j} \in W_{d j}$ . DFV value can be calculated by Eq. (7). $\begin{matrix} (7) & V (w_{i, j}) = docfreq (w_{i, j}) \times \sum_{d_{j} \in D} freq (w_{i, j}) \end{matrix}$ where $freq (w_{i, j})$ represents times a word $w_{i, j}$ appears in document $d_{j}$ . $docfreq (w_{i, j})$ represents times a word $w_{i, j}$ appears in all documents in D.

Words can be ordered by their DFV values. Some words, having the highest DFV value, i.e. $DFV = (w_{1}, w_{2}, \dots, w_{n})$ , which can be used to describe domain features.

We use the same corpus to test TF-IDF as that for experiments of testing DFV. The corpus marked as D, an individual document $d_{j} \in D$ , word in $d_{j}$ document $W_{d j}$ and an individual word $w_{i, j} \in W_{d j}$ . TF-IDF value of $W_{d j}$ can be calculated by Eq. (8). $\begin{matrix} (8) & \begin{matrix} TF - IDF (w_{i, j}) & = log \frac{|D|}{1 + docfreq (w_{i, j})} \\ \times \sum_{d_{j} \in D} freq (w_{i, j}) \end{matrix} \end{matrix}$ where $freq (w_{i, j})$ represents times a word $w_{i, j}$ appears in document $d_{j}$ . $docfreq (w_{i, j})$ represents times a word $w_{i, j}$ appears in all documents in D.

Usually, word in corpus contains multiple kind of noises. From the corpus, we traverse the whole corpus to wipe out noise words, which is one of the steps different from the traditional methods. After words in corpus have been filtered, DFV values are calculated to identify DFV element. The algorithm is described as below.

4.3. Document Classification Value (DCV)

Text categorization is defined as assigning predefined categories to text document [20]. Starting with a training set $D = (d_{1}, \dots, d_{n})$ of documents labeled with a category $c \in C$ , the task of text classification is to train the classification model $f : D \to C$ , $f (d) = c$ which is able to assign the correct class labels to a new document d of the domain. According to this objective, we design a method to evaluate our algorithm using text classification as a judgment. Document Classification Value (DCV) is defined by Eq. (9). $\begin{matrix} (9) & \begin{matrix} DCV (w_{i, j}) & = \sum_{W_{i, j} \in t_{j}} freq (w_{i, j}) \\ \times In (Length - Rank (w_{i, j}) + 1) \end{matrix} \end{matrix}$ where $w_{i, j}$ represents position of a word i in $DFV = (w_{1}, w_{2}, \dots, w_{n})$ . $Rank (w_{i, j})$ represents position of $w_{i, j}$ in DFV. Length represents length of DFV. $freq (w_{i, j})$ indicates times that word $w_{i, j}$ appears in document $t_{j}$ in the test set T. DCV generating from DFV is the judgment of text classification. The procedure of our evaluation method is described in Fig. 2.

Fig. 2.

The procedure of evaluation method.

The test set is composed of documents from domain A and domain B marked as A and B, respectively. After calculating DCV of DFV from domain A and domain B, i.e. $DCV_A$ and $DCV_B$ , they are compared to decide each document’s DFV predictive to the corresponding classification. The correct rate of document classification task is analyzed to evaluate our method. The algorithm is described as follows.

For one document, if DCV given by DFV of domain A is higher than DCV given by domain B, the document will be input into classification A, and vice versa. Accurate rate can be counted out. If both of DCV from DFV of domain A and domain B are both zero, the test result can be regarded as unrecognized value. The accurate rate AR and unrecognized rate UR compose the judgment of DFV model.

4.4. Effect of threshold selection on extraction results

No matter what domain basic terminology extraction or domain compound terminology extraction is, it is required to determine range of threshold, which has an important impact on the analysis. At the same time, for different domain, and even different scale of corpus in the same domain, threshold range may be different.

The main factors effecting threshold may include the following three aspects. 1) Corpus, including scale of corpus, domain-related degree of corpus itself and organizational structure of corpus; 2) Domain; 3) Selected seed terminology, including quantity of seed terminology, domain-related degree of seed terminology. While these factors change, range of the optimal threshold constantly changes. For the extraction of large scale of domain terminology, it is suggested to compare with accuracy under the different threshold range to roughly determine the threshold range through randomly extracting a small number of result sets, and then adjust the threshold range by the repeated operations.

5. Experiments

5.1. Data preprocessing

Many researches have done the work for word extracting and a number of methods have been proposed, which are indeed achieved satisfying results and have been used widely. However, there are still some challenges remain, especially for Chinese documents [6]. Chinese language is a typical para taxis language, without obvious separator, such as a space in English, between two words in a sentence. In order to extract words set from plain Chinese text, a sentence is spited into each word. For it, Viterbi HMM Model [5] is used in the paper. After splitting the word from sentence, each word will be marked according to their part of speech. And the classification standard from Peking University’s Daily Corpus POS tag set is used [19]. An example of this procedure is shown in Fig. 3.

Fig. 3.

Setting for document template.

5.2. Domain terminology extraction

Domain basic terminology extraction. Select the following five words in the domain transportation as of seed terminology in the first round of iteration, which is highway, road, traffic, lane and car. Use our TDE-I algorithm for iteration, making the result value be greater than or equal to 0.25 and the first 50 words as a threshold for filter. The result of the first round of iteration is shown in Fig. 4.

Fig. 4.

Candidate terminology generated in the field of transportation after the first round of iteration.

Add the words obtained in the first iteration into seed terminology and perform the second iteration. Similarly, taking the result value greater than or equal to 0.25 and the first 50 words as the threshold, repeat the iteration to obtain the results as shown in Fig. 5. The accuracy of the experimental results is 72.3%.

Fig. 5.

Candidate terminology generated in the field of transportation after the second round of iteration.

Fig. 6.

Candidate compound terminologies generated in transportation domain after multiple iterations.

Fig. 7.

Comparison in accuracy for transportation domain extraction between co-occurrence rate-based algorithm and TDE-I algorithm.

Domain compound terminology extraction. The first 50 words are used as a filter to be a threshold range. Candidate compound terminologies generated in transportation domain after multiple iterations are shown in Fig. 6. The results are greater than or equal to 0.25. The accuracy of the experimental results is 78%.

Algorithm comparison. We also conduct experiment to make comparison between co-occurrence rate-based algorithm and our TDE-I algorithm. We make terminology extraction from 863 transportation-related corpus of the Intelligent Transportation Network and calculate their accuracy, respectively. The accuracy of the two algorithms and their comparison are shown in Fig. 7.

As we can see from Fig. 7, the key terms extracted from the co-occurrence rate-based algorithm may contain more invalid words, such as the terms of reform, development in Chinese. Number of occurrences that they combine with seed terminology are also more, and also frequency appears in the traffic articles is also larger; while our algorithm reduces the weight of the domain words by means of new word discovery, part of speech filtering and relative co-occurrence rate, and also the accuracy of the results is improved.

5.3. Evaluation method

In order to test our evaluation method, in our experiments, we construct corpus by collecting 10,377 pieces of documents as a sample set of domain technology from the RSS data source (http://cn.engadget.com), and 2,333 pieces of documents as a sample of domain health from the RSS source (http://nhzy.org). Also, we collect 6,778 documents from the RSS source (http://cn.Beta.com) as foreign testing set.

DFV value. We construct the corpus of domain health and domain technology. Applying the proposed DFV algorithm to the corpus, we can obtain the DFV results of domain health and domain technology, respectively, as shown in Fig. 8. The default value of length of DFV is 100.

Fig. 8.

The result of DFV algorithm applying to a Chinese corpus.

The distribution of DFV value of domain health and domain technology is shown in Fig. 9.

Fig. 9.

DFV value distribution.

By analyzing the distribution of DFV value from both domains, it is shown that it is similar to logarithmic distribution, which satisfying with the distribution defined in DCV.

DCV value. We randomly select 300 pieces of document from both domain health and domain technology from our corpus as a sample data set. We also count DCV value of $Health_DFV$ and $Tech_DFV$ of each document. The normalized value of DCV from domain health and domain technology are used to decide classification of each document, which is shown in Fig. 10.

Fig. 10.

Results of document classification from corpus.

From Fig. 10, we can conclude that the DFV extracted from one domain shows a great distinction from another domain documents, which proves that the DFV is suitable to extract domain features. Besides, we also conduct experiment about applying DFVs to classify documents from foreign documents, the result is shown in Fig. 11.

Fig. 11.

Document classification from foreign data set.

From Fig. 11, DFV indeed have distinction between different domains, words in DFV can be recognized as domain terminologies. One more result, the length of DFV is determined, as shown in Fig. 12.

Fig. 12.

Experiment result of determination of the length of DFV.

At beginning, as the length increases, the recognition accuracy begins to increase. However, at the length of 50, it reaches the top. From Fig. 12, DFV of length 50 is the most suitable one, which has the biggest domain distinction.

Comparison analysis. In our experiments, by calculating TF-IDF value, we also extract the vector, which has the length of 50, the same with that of DFV. We compare our DFV model with TF-IDF using text classification evaluation experiments, as shown in Table 2 and Table 3.

Table 2

Comparison with documents from corpus

DFV		TF-IDF

Accuracy rate	Unrecognized rate	Accuracy rate	Unrecognized rate
90.84%	0%	41.82%	45%
91.33%	0%	39.56%	53.67%
89.34%	0%	43.22%	48.33%
90.83%	0%	38.14%	56.67%
90.17%	0%	41.46%	47.33%

Table 3

Comparison with documents from foreign data set

DFV		TF-IDF

Accuracy rate	Unrecognized rate	Accuracy rate	Unrecognized rate
84.33%	0%	55.56%	73%
83.33%	0%	48.63%	76%
85%	0%	47.87%	76.33%
88%	0%	61.44%	72.33%
85.67%	0%	56.63%	72.33%

From the comparison results in Table 2 and Table 3, traditional method TF-IDF has obvious high error rate and unrecognized rate; while our DFV is better in both indexes.

5.4. Threshold effect

Effect of extracting domain terminology in the same domain. Taking transportation domain as an example, we use 863 articles in Intelligent Transportation Network and 1,277 articles China Intelligent Transportation Network as the corpus, respectively. Through TDE-I algorithm and seed terminology of transportation domain, the results of correct rate after the first round iteration are shown in Table 4 and Table 5, when using different threshold. The items with the highest accuracy are marked in gray in the tables. The results shows that, as the corpus is different, the threshold of the optimal results may not be the same under the same situation of seed terminology and using the same algorithm. It is concluded that the optimal threshold range is not fixed for the same domain.

Table 4
Extraction results of accuracy using TDE-I under Intelligent Transportation Network

Table 5

Extraction results of accuracy using TDE-I under China Intelligent Transportation Network

Taking transportation domain as an example, we use 863 articles as corpus under Intelligent Transportation Network. Applying for TDE-I algorithm, through seed terminology in the transportation domain mentioned in the previous section, but a different threshold range, the results of the accuracy after the first round iteration is shown in Table 4 (The highest accuracy is marked with Grey).

For corpus of China Intelligent Transportation Network, we conduct 1,277 articles as corpus. Using the improved extraction algorithm, and also using the seed terminology of transportation domain mentioned in the previous section, but different threshold range, the results of correct rate after the first round iteration are shown in Table 5 (The highest accuracy is marked with Grey).

From the above comparison results, we can see, as the corpus is different, under the situation of the same algorithm and seed terminology, the threshold range of the optimal result is not the same. It is concluded that the optimal threshold range may not fixed for the same domain.

Effect of extracting domain terminology in the different domain. Taking financial domain and transportation domain as examples, 6,000 economic and financial related articles in Oriental Fortune Network and 863 industry news in Intelligent Transportation Network are used as corpus, respectively, comparison results as shown in Table 6 and Table 7, different domain and different scale of corpus, range of the highest accuracy may change.

Table 6

Extraction results in financial domain

Table 7

Extraction results in transportation domain

From the above comparison results, we can see, the same as the above conclusion of effect of different thresholds in the same domain, range with the highest accuracy may change when both domain and scale of corpus are different.

Threshold range selection. Besides financial domain and transportation domain, we also make analysis on some other fields of education, medicine, chemistry, physics, and literature and so on. Although the minimum result value is set in the range of 0.2 to 0.25, we cannot guarantee extraction accuracy of domain terminology must be the biggest. But accuracy rate is relatively high, and possibility is also larger. The selection of the optimal threshold range is not fixed.

6. Conclusions

The construction of knowledge graph, as a massive systemic engineering, requires a serials of tasks, including domain terminology extraction, ontology construction, and relationship construction. Domain terminology extraction is the key and primary step. However, the existing traditional methods are time-consuming and laborious, as well as low accuracy rate and much invalid word. In order to deal with these problems, we propose a new domain terminology method which can complete the task automatically.

Our contributions can be summarized as follows.

For the problem of low accuracy with invalid terms, and poor adaption to high knowledge refreshing rate in domain terminology recognition and extraction, we propose our improved Domain Term Extraction-Improvement (TDE-I) based on relative co-occurrence rate. The experiment results show that our algorithm is better than the traditional algorithm.

There is not yet a uniform standard for evaluating domain terminology extraction algorithm. We present a Document Classification Value (DCV) method on the basis of calculating Domain Feature Vector (DFV) value, which can implement a judgment to evaluate extraction accuracy according to the evaluation indexes of accuracy rate, unrecognized rate. The experiment results demonstrate that the DCV method is effective.

As small-scale corpus may contain very limited number of domain terminologies, it is usually necessary to consider the method which can handle large-scale corpus for terminology extraction. However, the difficulty will be met to handle some corpus beyond memory. A feasible solution is to divide large corpus into several sub-corpus. We can import the divided results into memory and build an index file, and then make processing on them. If the size of corpus is still larger than memory after division, the corpus will continue to be divided until it can be imported into the memory once. This method can effectively avoid the disadvantages frequent disk I/O operations due to multiple reads of corpus.

Further work includes as follows. The main limitation of the current study is the lack of thorough experiments with other corpus of some other languages. Nevertheless, since the objective of this paper is to propose the DFV model, it remains as natural future work the experimentation of our proposal to a statistically significant set of corpus of different languages and related to more specific domains.

Footnotes

Acknowledgements

This work is supported by Shanghai Academy of Spaceflight Technology (no. SAST2016082) and Open Research Fund of the Academy of Satellite Application (no. 2014_CXJJ-YG_02).

References

Amit, Introducing the knowledge graph: Things, not strings, Internet, 2012, available from: http://googleblog.Blogspot.co.uk/2012/05/introducing-knowledge-graph-things-not.html.

Fukushige and

Noguchi, Statistical and linguistic approaches to automatic term recognition: NTCIR experiments at Matsushita, Terminology 6(2) (2001), 257–286. doi:10.1075/term.6.2.08fuk.

Homayouni,

Heinrich,

Wei et al., Gene clustering by latent semantic indexing of MEDLINE abstracts, Bioinformatics 21(1) (2004), 104–115. doi:10.1093/bioinformatics/bth464.

Ittoo and

Bouma, Minimally-supervised extraction of domain-specific part-whole relations using Wikipedia as knowledge-base, Data & Knowledge Engineering 85 (2013), 57–79. doi:10.1016/j.datak.2012.06.004.

K.F.

Lee, Automatic speech recognition: The development of the SPHINX system, in: Automatic Speech Recognition, Vol. 62, Springer Science and Business Media Press, 1988, 207 pages.

Li and

Zhang, Keyword extraction based on TF/IDF for Chinese news document, Wuhan University Journal of Natural Sciences 12 (2007), 917–921. doi:10.1007/s11859-007-0038-4.

Lin, Mausam and

Etzioni, Entity linking at web scale, in: The Joint Workshop on Automatic Knowledge Base Construction and Web-Scale Knowledge Extraction, ACM Press, Montreal, Canada, 2012, pp. 84–88.

Lopes,

Fernandes and

Vieira, Domain term relevance through TF-DCF, in: Proceedings on the International Conference on Artificial Intelligence (ICAI), CSREA Press, Las Vegas, USA, 2012, pp. 1001–1007.

H.P.

Luhn, The automatic creation of literature abstracts, IBM Journal of research and development 2(2) (1958), 159–165. doi:10.1147/rd.22.0159.

10.

NewScientist, Why Google will soon answer your questions directly, Internet, May 30, 2012, available from: http://www.newscientist.com/article/mg21428676.400-why-google-will-soon-answer-your-questions-directly.html#.VP-zK-Gb8th.

11.

M.T.

Pazienza,

Pennacchiotti and

F.M.

Zanzotto, Terminology extraction: An analysis of linguistic and statistical approaches, in: Knowledge Mining, Springer-Verlag, Berlin, Heidelberg, 2005, pp. 255–279. doi:10.1007/3-540-32394-5_20.

12.

Pujara,

Miao and

Getoor, Ontology-aware partitioning for knowledge graph identification, in: Proceedings of Automated Knowledge Base Construction, ACM Press, Atlanta, Georgia, 2013, pp. 19–24.

13.

Quirchmayr,

Paech,

Kohl et al., Semi-automatic rule-based domain terminology and software feature-relevant information extraction from natural language user manuals, in: Empirical Software Engineering, Springer US, 2018, pp. 1–54.

14.

Saedi,

Artetxe,

Labaka et al., Adding syntactic structure to bilingual terminology for improved domain adaptation, in: Proceedings of the 2nd Deep Machine Translation Workshop, Lisbon, Portugal, 2016, pp. 39–46.

15.

Shinnou,

Takeuchi and

Collier, Bio-medical term extraction with morpho-syntactic rules on simple rule language, Ieice Technical Report 109(5) (2009), 788–797.

16.

Steiner,

Verborgh,

Troncy et al., Adding realtime coverage to the Google knowledge graph, in: Proceedings of the International Conference on Posters and Demonstrations Track, ACM Press, Boston, USA, 2012, pp. 65–68.

17.

Suchanek and

Weikum, Knowledge harvesting in the bigdata era, in: The ACM SIGMOD International Conference on Management of Data, ACM Press, New York, USA, 2013, pp. 933–938.

18.

Sullivan, Google launches knowledge graph to provide answers, not just links, Internet, May 16, 2012, available from: http://searchengineland.com/google-launchesknowledge-graph-121585.

19.

Yu,

Duan,

Zhu et al., The basic processing of contemporary Chinese corpus at Peking University specification, Chinese Journal of Information Science 16(5) (2002), 51–66.

20.

Zhang,

Yoshida and

Tang, A comparative study of TF*IDF, LSI and multi-words for text classification, Expert Systems with Applications 38 (2011), 2758–2765. doi:10.1016/j.eswa.2010.08.066.

21.

Zhou and

Zhang, NLPIR: A theoretical framework for applying natural language processing to information retrieval, Journal of the Association for Information Science and Technology 54 (2003), 115–123.