Learning time-sensitive domain ontology from scientific papers with a hybrid learning method

Abstract

Large numbers of available scientific papers makes the research of ontology construction an attractive application area. However, there are two shortcomings for most current ontology construction approaches. First, implicit time properties of domain concepts are rarely taken into account in current approaches. Second, current automatic concept relation extraction methods mainly rely on the local context information that surrounds current considered concepts. These two problems prevent most current ontology construction methods from being employed to their full potential. To tackle these problems, we propose a hybrid learning method to integrate concepts’ global information and human experts’ knowledge together into ontology construction, among which concepts’ temporal attributes are taken into account. Our method first divides each concept into four time periods according to their attribution distribution on a time axis. Then global time-related attributions are collected for each concept. Finally, concept relations are extracted with a hybrid learning method. We evaluated our method by testing it on Chinese academic papers. It outperformed a baseline system based on only hierarchical concept relations, showing the effectiveness of our approach.

Keywords

domain ontology life cycle division relation assignment time sensitive

1. Introduction

Scientific papers (such as patent documents and academic papers) are a kind of high-volume and high-value information asset. Most scientific papers contain a rich source of information like titles, authors, institutions, journals, keywords, abstracts and contents. Such information makes it possible for researchers to mine some large scale and practical domain ontologies for following two reasons. First, most of academic journals are usually related to some specific domains, thus one can easily obtain large-scale domain-specific data by just collecting those scientific papers that are published in some predefined journals. Second, most keywords in these scientific papers are good domain concepts, thus one can skip the error-prone concept identification process that is always perplexing researchers in traditional domain ontology construction methods. Recently, such domain ontology construction attempts have been carried out by researchers. For example, Ren [1, 2] has proposed a simple method to construct a domain ontology from scientific papers. In his method, some keywords among scientific papers are selected as domain concepts and concepts’ hierarchical relations are assigned based on concepts’ co-occurrence information and frequency information. Similar research work has also been done by Brijesh and Pushpark [3]. However, most of these related ontology construction methods have the following two shortcomings.

First of all, to the best of our knowledge, previous research has rarely taken into account the time properties of domain concepts. There are two observations to make here. First of all, not all domain concepts are constant: over time, some concepts will vanish, and some concepts’ attributions will change. Second, even for those constant concepts, it will take some time for them to fully display their attributions. This means that, if we assign every concept a life cycle (this life cycle usually can be divided into arising period, expansion period, stability period and optional extinction period), the concept usually displays different attributions in different periods. Overlooking these time properties can lead to the following problems:

It will increase concept confusion because of the failure to detect concepts’ vicissitudes. For example, the concept ‘natural language understanding’ (NLU for short) in the the IT domain is seldom mentioned in Chinese academic papers now. This can be clearly seen from Figure 1, where we show the respective related papers over the past 30 years (from 1979 to 2010).¹ Currently, when one says ‘natural language understanding’ in China academia, one is more likely to be referring to the concept ‘natural language processing’ (NLP for short). In such a case, it will be difficult for us to give a correct interpretation of ‘natural language understanding’ if we cannot detect this concept’s change. Thus it is necessary for the domain ontology construction method to detect such changes in concepts in a timely manner, so that more accurate information about a concept can be provided to users. (In Figure 1, including the following Figures 2 and 3, ‘related topic’ refers to the number of keywords that co-occur with the current concept. ‘Related journals’ refers to the number of journals containing the current concept at least one paper’s keyword list. ‘Related researchers’ refers to the number of researchers who published at least one paper with a keyword list containing the current concept. ‘Related papers’ refers to the number of papers whose keyword list contains the current concept. )

It will be more prone to make mistakes when assigning relations for such kind of concept pairs whose concepts are at different periods of their respective life cycles. This is because most current relation assignment methods can be regarded as a kind of feature-based method in which a decision is made mainly based on concepts’ attributions. However, if two concepts are at different periods of their respective life cycles, the integrity of their possessing attributions would be different and asymmetrical. In such case, these feature-based relation assignment methods would not reflect the real situation of the current concept pair. Thus the probability of causing relation assignment errors would increase accordingly. The concepts ‘machine translation’ and ‘sentiment analysis’ are a good illustration of this observation because they have many common attributions and are likely to be assigned a relation. Figures 2 and 3 also demonstrate some of the statistics regarding their attributions over the past 30 years from 1979 to 2010, based on the data downloaded from https://www-cnki-net.web.bisu.edu.cn/. These figures clearly show huge differences in their attributions over time. However, when assigning relations to them, most of the current methods are prone to assigning them a hierarchical relation from ‘machine translation’ to ‘sentiment analysis’ because ‘sentiment analysis’ has a smaller attribution collection and a smaller attribution statistics. However, both of them are sub-tasks of ‘natural language processing’. Note that the range of the x-axis in Figure 2 is different from that in Figures 1 and 3. This is because ‘sentiment analysis’ is a new emerging concept and it appeared in published academic papers from 2007, which is far later than ‘machine translation’, ‘natural language processing’, or ‘natural language understanding’.

Figure 1.

An comparison of NLP and NLU.

Figure 2.

The concept ‘sentiment analysis’ over time.

Figure 3.

The concept ‘machine translation’ over time.

Second, most of the current automatic concept relation extraction methods are verb-driven and mainly rely on the local context information that surrounds current considered concepts. This will greatly limit the number of possible relations. If there are no proper verbs identified in a local content (often sentences) or if two concepts do not co-occur in a local content, no proper relationships will be found. A simple example can demonstrate this case. For example, taking the two concepts ‘classifier’ and ‘support vector machine’ as an example, it is clearly that ‘support vector machine’ is a kind of ‘classifier’, thus a ‘kind of’ relationship should be assigned between them. However, if there is no sentence like ‘support vector machine…is a kind of…classifier’, their relationship will be never detected by most current relationship extraction methods.

Both of these two shortcomings in current ontology construction methods should be tackled to obtain high performance domain ontologies. To tackle these shortcomings, we propose a hybrid learning method to integrate concepts’ global information and human expert knowledge into ontology construction, among which concepts’ temporal attributes are taken into account. Our method first divides each concept into four time periods according to its attribution distribution in time axis. Then global time-related attributions are collected for each concept. Finally, concept relations are extracted with a hybrid learning method.

2. Related work

The development of the Semantic Web makes ontology construction a hot issue. Until now, many researchers have proposed many efficient automatic ontology construction methods [1 –43]. According to the concept relation assignment methods used, most of the current automatic ontology construction methods can be classified into three types.

The first one is to extract concept relations with the help of some general purpose resources. This kind of method usually consists of two phases. The first phase will extract concepts from some annotated corpus with the help of machine learning techniques and natural language processing techniques. The second phase will assign relations for these concepts with the help of some general resources. In this phase, WordNet is usually selected as the needed resource. For example, the famous domain ontology tool OntoLearn [4] first extracts the domain terminologies from some available documents mainly based on two measures that are called Domain Relevance and Domain Consensus. Then, complex domain terms are semantically interpreted and arranged in a hierarchical fashion with the help of WordNet. Finally, WordNet is trimmed and enriched with the detected domain concepts. Similar work can also be found in Boonchom and Soonthornphisaj [5], in which they constructed a Thai legal ontology with the help of a terminology resource. In their method, they first automatically generate a seed ontology and then expand this ontology using a Thai legal terminology dictionary named TLlexicon. Many domain ontology construction methods can be classified into this category, such as the research work in Navigli et al. [6], OntoLearn [7] and Text2Onto ([8], which is an improved version of OntoLearn).

The second one is rule-based relation assignment methods. For example, research work such as Ruiz-Martínez et al. [9] is based on rule-based approaches to relation extraction.

The third one is pattern-based relation assignment methods. Currently, this is a kind of dominant relation extraction method, and many methods can be classified as this kind. This kind of method is mainly based on the fact that, in natural language, relationships between concepts are usually associated with verbs [10]. Thus a number of systems for learning relationships have been proposed based on the extraction and identification of verbs. Usually, the final learned relationships will be a tri-tuple set. Each item of this set will have a format like ‘concept_m verb_i concept_n’, which means that concept_m has a relationship with concept_n decided by the verb verb_i. For example, Ochoa et al. [11] proposed a semantic role labelling-based framework for learning ontologies from Spanish documents. In their method, they first identify the main verb in an input sentence; then they look for concepts on the right and left side of the main verb; finally the corresponding relationship between the left concept and the right concept are determined by the main verb. To further determine the relation type, a set of linguistic patterns along with the semantic role labelling information are used. Other similar work can be seen in the literature [12 –20].

However, all of these ontology construction methods have the following two shortcomings. First, to the best of our knowledge, among these ontology construction methods, researchers have seldom noticed the time property hidden in concepts, which prevents many traditional approaches from being employed to their full potential. Although Zou and Park [21] discussed the temporal information in ontology construction, their work aims to construct an ontology about the ancient Chinese lunisolar calendar. That is to say, they did not discuss the issue that the concepts’ attributions will change over time either. Second, to the best of our knowledge, among these ontology construction methods, researchers have seldom used the global information during concept relation extraction. Most of these researchers only focus on the content within a sentence. That is to say, when only two concepts appear in one sentence, and also are adjacent to one verb, they will be assigned some relations. This will greatly limit the number of possible relations. For example, taking two concepts ‘classifier’ and ‘support vector machine’, as an example, it is clear that ‘support vector machine’ is a kind of ‘classifier’, thus a ‘kind of’ relationship should be assigned between them. However, if there is no such sentence as ‘support vector machine…is a kind of…classifier’, their relationship would be never detected by most current relationship extraction methods. Although the first two kinds of relation extraction methods seem to be able to solve this issue, the problem is that, for the first kind of method, not all language can find a proper general purpose resource like WordNet, and for the second kind of method, because of the high construction cost and its time-consuming nature, it is impossible to cover all of the possible relationships with some human-designed rules. It is worth noting that Hou et al. [17] proposed a graph-based approach for automatic construction of domain ontology. In their method, they use content information within a document. Specially, each document in the collection is represented by a graph in their method. After the generation of document graphs, a method called random walk term weighting is employed to estimate the relevance of the information of a term to the corpus from both local and global perspectives. Next, the Markov Clustering algorithm is used to disambiguate terms with different meanings and group similar terms to produce concepts. Next, an improved gSpan algorithm constrained by both vertices and informativeness is exploited to find arbitrary latent relations among these concepts. In fact, their method can alleviate the considered issue to some extent. However, their method still cannot assign relations for those concepts that do not appear in one document, even though these concepts may still have some potential relationships.

It should also be noted that Ren [1, 2] also used a graph-based method for domain ontology construction, and in their method, all the concepts along with their relationships are considered in the whole domain, so the global information is used in their methods. However, their concept relationships are extracted only based on concepts’ co-occurrence and frequency information. We think that more useful global information should be used during concept relationship extraction, so there is much space for improvement.

3. Our method

We aim to propose a new method to construct a domain ontology from scientific papers which can solve the two problems described above effectively. The proposed method is also a graph-based domain ontology construction method. The basic ideas are as follows. First, every concept can be represented as a kind of global time–attribution chain. Second, for any two concepts, their relations can be determined mainly according to the similarity between their time–attribution chains. This kind of similarity can be obtained with a hybrid method. Specifically, we think that every domain concept has a life cycle that can be divided into an arising period, an expansion period, a stability period and an extinction period. Generally speaking, the earlier period a concept is in, the more incomplete attributions it will possess. Thus if two concepts are in different periods, their similarity value should be adjusted accordingly. In the following, we first give some definitions that are used in this paper, and then introduce our method in detail.

3.1. Definition description

3.1.1. Definition 1 (concept attribution)

A domain concept c_i’s attribution Attr(c_i) is defined as a five-tuple <TS(c_i), RS(c_i), JS(c_i), OS(c_i), KS(c_i)>, in which every item is a set that contains some information related to c_i. Specifically, TS(c_i) is a set composed of the titles of some papers where c_i is selected as one of their keywords, RS(c_i) is a set composed of the researchers who have published at least one paper that c_i is selected as a keyword, JS(c_i) is a set composed of the names of some journals where c_i is selected as a keyword by at least one paper published in these journals, OS(c_i) is a set composed of the names of some research organizations where c_i is selected as a keyword by at least one paper published by at least one researcher in this organization, and KS(c_i) is a set composed of the keywords that have co-occurred with c_i at least one time.

3.1.2. Definition 2 (concept moment attribution)

A domain concept c_i’s moment attribution MomAttr(c_i, t_j) is defined as its attribution at a specific time point t_j. In fact, a concept c_i’s moment attribution is also a five-tuple which can be denoted as <TS(c_i, t_j), RS(c_i, t_j), JS(c_i, t_j), OS(c_i, t_j), KS(c_i, t_j)>.

3.1.3. Definition 3 (concept time accumulation attribution)

A domain concept c_i’s time accumulation attribution AccuAttr(c_i, t_m, _n) is defined as a five-tuple, in which every item is a union of the corresponding set in c_i’s moment attribution over the time period t_m, _n(n > m). Specifically, c_i’s time accumulation attribution AccuAttr(c_i, t_m, _n) is formally defined in equation (1):

AccuAttr (c_{i}, t_{m, n}) = < \sum_{t = t_{m}}^{t = t_{n}} \cup TS (c_{i}, t), \sum_{t = t_{m}}^{t = t_{n}} \cup RS (c_{i}, t), \sum_{t = t_{m}}^{t = t_{n}} \cup JS (c_{i}, t), \sum_{t = t_{m}}^{t = t_{n}} \cup OS (c_{i}, t), \sum_{t = t_{m}}^{t = t_{n}} \cup KS (c_{i}, t) >

(1)

3.1.4. Definition 4 (attribution variation rate)

A domain concept c_i’s attribution variation rate AttrVar(c_i, t_m, _n) is defined as a five-tuple, in which every item is a number variation in c_i’s corresponding attribution tuple over a given period of time t_m, _n(n > m). Specifically, c_i’s attribution’s variation rate is formally defined in equation (2):

\begin{matrix} AttrVar (c_{i}, t_{m, n}) = & < \frac{| TS (c_{i}, t_{n}) | - | TS (c_{i}, t_{m}) |}{t_{n} - t_{m}}, \frac{| RS (c_{i}, t_{n}) | - | RS (c_{i}, t_{m}) |}{t_{n} - t_{m}}, \frac{| JS (c_{i}, t_{n}) | - | JS (c_{i}, t_{m}) |}{t_{n} - t_{m}}, \\ \frac{| OS (c_{i}, t_{n}) | - | OS (c_{i}, t_{m}) |}{t_{n} - t_{m}}, \frac{| KS (c_{i}, t_{n}) | - | KS (c_{i}, t_{m}) |}{t_{n} - t_{m}} > \end{matrix}

(2)

3.1.5. Definition 5 (concept life cycle)

A domain concept c_i’s life cycle Cyc(c_i) is defined as a period of time t_m, _n from the time of c_i’s first appearance in a scientific paper’s keyword list (denoted as t_m) to the time of its completely disappearance from any scientific papers’ keyword lists (denoted as t_n). Usually, a concept’s life cycle can be divided into following four stages:

Arising period. This refers to a period starting from the first appearance of a specific domain concept, and a concept’s attribution will expressed in a slowly rising way in this period.

Expansion period. This refers to a period following the arising period. In this period, a concept’s attribution will expressed in a sharply rising way.

Stability period. This refers to a period following the expansion period. In this period, a concept’s attribution will expressed in a steady way.

Extinction period. This refers to a period following the stability period. In this period, a concept’s attribution will expressed in a rapidly falling way.

3.1.6. Definition 6 (directly adjacent concepts)

A concept c_m is directly adjacent to a concept c_n if c_m and c_n appear in the keyword list of a same paper.

3.1.7. Definition 7 (indirectly adjacent concepts)

A concept c_m is indirectly adjacent to another concept c_n if the following two conditions are satisfied:

c_m and c_n are not directly adjacent;

c_m and c_n appear in the keyword list of paper p_m and p_n respectively, and p_m and p_n have at least one common author.

3.1.8. Definition 8 (adjacent concepts)

A concept c_m is adjacent to a concept c_n if they are either directly adjacent concepts or indirectly adjacent concepts.

3.1.9. Definition 9 (adjacent strength)

We define the adjacent strength as a metric that is used to describe the reliability of linking two concepts in a domain ontology graph. We define two kinds of adjacent strength according to the adjacent type of two concepts.

First of all, if two concepts c_m and c_n are directly adjacent, we define their directly adjacent strength DirAdjStr(c_m, c_n) as shown in formula (3), in which k is the co-occurrence number for c_m and c_n, and d_k(c_n) is the order of c_n in the kth keyword list. Here we take the concepts’ appearance orders in keyword lists into consideration mainly because of the following fact: sometimes these orders can discover a kind of ‘generalized-specialized’ relations among concepts.²

DirAdjStr (c_{m}, c_{n}) = \sum_{k} (1 + \frac{1}{d_{k} (c_{n}) - d_{k} (c_{m})})

(3)

Second, if two concepts c_m and c_n are indirectly adjacent, and the number of their related paper pair (p_m, p_n) is k, we define their indirectly adjacent strength IndAdjStr(c_m, c_n) as ϵ*k, and ϵ is an adjusting factor.

Finally, for two concepts c_m and c_n, we define their adjacent strength AdjStr(c_m, c_n)as the sum of DirAdjStr(c_m, c_n) and IndAdjStr(c_m, c_n).

3.1.10. Definition 10 (concept degree)

A concept c_m’s degree deg(c_m) is defined with formula (4):

\deg (c_{m}) = \sum_{k} AdjStr (c_{m}, c_{k})

(4)

3.2. Method description

It is naturally to use a directed graph to represent a domain ontology. In this graph, every node represents a domain concept and every directed edge represents a specific relation from its starting concept to its ending concept. Thus the procedure of domain ontology construction is also the procedure of constructing such a directed graph. There are two phases in our method to construct such a domain ontology graph. The first phase is a preprocessing phase and the second is the domain ontology graph construction phase.

3.2.1. Preprocessing phase

Follow the work in Ren [1, 2], we also take all of the keywords in academic papers as the candidates for domain concepts. In this phase, we will finish the following two tasks. First of all, we will obtain a moment attribution chain for each domain concept. This moment attribution chain can be denoted as {a_i₁, a_i₂, …, a_in}, in which every item a_ik represent the concept c_i’s kth moment attribution MomAttr(c_i, t_k) and n is the number of years that c_i appears in some papers. The time unit used here is years. Second, we will find all of the adjacent concept pairs along with their corresponding adjacent strengths.

3.2.2. Domain ontology graph construction

When the domain concept candidates have been determined in the first phase, the key of the domain ontology graph construction is converted into the task of assigning proper relations to these domain concepts. Here we design a two-step method for this relation assignment task. The first step is to divide the life cycle into proper periods for each concept. The second step is a relation assignment process.

3.2.3. Step 1: life cycle division

The life cycle division task is to divide a concept’s life cycle into proper periods. According to these periods’ definitions, it can be seen that this division is mainly determined by a concept’s attribution variation rate and the division process is somewhat like a process of finding jump points on this concept’s attribution variation rate curve. However, there are two points that should be noted. First of all, we require that the number of jump points should be not greater than 3 from a global perspective for a concept’s attribution variation rate curve. This requirement is mainly used to simplify the tackled issue. One can easily see that the real fact may be more complex than expected. This is because, except for the arising period, all the other periods may appear multiple times in a concept’s attribution variation rate curve. For example, it is possible that we can have a life cycle involving arising, expansion, stability and arising again, and so on. However, if we look a concept’s attribution variation rate curve from a global perspective, all of these inter-reduplicate periods can be viewed as noise points. This will not affect the performance of our final constructed ontology for the reason that the original motivation for introducing the life cycle is to evaluate the completeness of a concept’s attributions. Second, every variation rate is in fact a five-tuple, so the points in a concept attribution’s variation rate curve should be a combination of these tuples. Based on these analyses, we designed a life cycle division algorithm that is described in Figure 4.

Figure 4.

Algorithm for life cycle division.

In the above algorithm, cur is a counter that is used to denote the current division period, and div(a_it) is a function that is used to determine the type of current time point t. Specifically, div(a_it) is computed according to the rules shown in Table 1, in which ‘div(a_it) = 1’ denotes t as a cutting point for the arising and expansion periods, ‘div(a_it) = 2’ denotes t as a cutting point for the expansion and stability periods, ‘div(a_it) = 3’ denotes t as a cutting point for the stability and extinction periods and ‘div(a_it) = 0’ denotes t as an inner point among some periods.

Table 1.

Decision rules for life cycle division.

	Decision rules
div(a_it) = 1	(1) a_i,_t≥δa_i, _t-1; (2) a_i, _t+1≥δa_i,_t; (3) a_i, _t+1≥δa_i, _t-1; (4) t-rst[cur-*1]≥2;
div(a_it) = 2	(1) cur = 1; (2) a_i,_t≥δa_i, _t-1; (3) a_i, _t+1≤γa_i,_t; (4) a_i, _t+1≥δa_i, _t-1; (5) t-rst[cur-*1]≥2;
div(a_it) = 3	(1) a_i,_t≤δa_i, _t-1; (2) a_i, _t+1≤δa_i,_t; (3) a_i, _t+1≤δa_i, _t-1; (4) t-rst[cur-*1]≥2;
div(a_it) = 0	Others

In fact, we designed these decision rules mainly based on the following two factors: the attribution variation rate on the cutting point between two adjacent periods and the time interval between two adjacent periods.

3.2.4. Step 2: relation assignment

In our method, only the adjacent (including directly adjacent and indirectly adjacent) concepts can be assigned relations. In this paper, we mainly focus on the following five kinds of concept relations: hierarchical relation, equivalence relation, has-a relation, part-of relation and kind-of relation. Based on the descriptions in Wikipedia (http://en.wikipedia.org/), we give these relations definitions as follows:

Hierarchical relation is a basic relationship in a domain ontology, in which concepts are represented as being ‘above’ or ‘below’ one another. In fact, every directed edge can be seen as a hierarchical relation whose starting concept is a superordinate concept and ending concept is a subordinate concept.

Equivalence relation is a relationship in which the related concepts have high attribution similarities. For example, we think there is an equivalence relation between the concept ‘natural language processing’ and the concept ‘natural language understanding’.

Has-a relation is a relationship where one concept (often called the composited concept) belongs to (is a part or member of) another concept (called the composite concept).

Part-of relation is just the inverse version of the has-a relationship. Therefore we can easily add this relationship to the has-a relationship by adding edges in the other direction.

Kind-of relation is a relationship where one concept is a specific example of another concept.

To assign proper relations on adjacent concept pairs, we design a hybrid method that combines human expert knowledge and a machine learning scheme together. Our basic ideas are as follows.

First, human expert’s knowledge used for deciding the relation of two concepts is not limited by the size of content windows and normally the decision results have a high precision within a small sample scale. Usually, human experts decide two concepts’ relation mainly based on their own experiences, which is a kind of global information related to current considered concepts. Thus those rule-based concept relation learning methods have a natural advantage in using global information to decide two concepts’ relation compared with other concept relation learning methods. However, as we pointed out previously, the problem is that it is difficult to cover all of the possible relationships with some limited human designed rules. On the other hand, we notice that, if we take all of the collected global information about two considered concepts as features related them, the traditional concept relation assignment problem is naturally converted into an automatic classification problem that has been well studied in recent years. A classification model can effectively combine different kinds of features so that an optimal result can be obtained for each input sample. Thus, this kind of automatic classification method always outperforms those rule-based methods. In addition, one can easily add all kinds of features to these classification methods regardless of global features or local features. Thus, this kind of classification method is also not limited by the size of content windows during concept relation learning. However, one must provide enough labelled training samples (e.g. some concept pairs whose relations have been correctly labelled) for learning a strong classification model to obtain a better classification result. Usually, the more training samples provided, the better the classification model will be, even in a far larger sample scale compared with provided training samples. From the above analysis we can see that both kinds of learning methods have their own advantages and shortcomings: rule-based learning methods can provide more precise decision results but only suitable for a small sample scale; classification-based learning methods can provide good decision results even in a larger sample scale but need labelled samples for model training. Therefore, it is natural to combine them in a simple way to obtain a better concept relation learning result: we can first ask some human experts to label a small number of concept relations; then these labelled relations will be taken as training samples to train an initial classification model; with the trained model, we can learn more concept relations; then from these learned concept relations, we carefully select some of them to be checked by human experts, and these checked samples along with previous training samples will be taken as new training samples to retrain a new classification model. This iteration can be terminated in a flexible way: iteration times, time spent, current precision obtained, and so on. Specifically, our concept learning method is shown in Figure 5.

Figure 5.

Our hybrid concept relation learning method

From Figure 5 we can clearly see that the following three sub-tasks that should be solved carefully. First of all, how to make the human labelled relations in a unified way; second, how to train a proper classification model; and third, how to select a proper set for labelling and relabelling.

For the first sub-task, we design a heuristic learning method that combines human experts’ knowledge in a proper way. For the second sub-task, we use a classification-based learning method to combine global feature information and local feature information. For the third sub-task, we design a simple random selection method for labelling and a classification probability-based selection method for relabelling. We will give detail description of our solution for these sub-tasks in subsequent sections.

3.2.5. Heuristic learning method

In this heuristic learning method, we obtain the needed relations by some heuristic rules in which expert knowledge is integrated. Specifically, for every two adjacent concepts, we define following three possible operations for them: link, abandon and shift.

Link operation means that we should add a directed edge for current two concepts and their relation should be further assigned.

Abandon operation means that we should do nothing to the current two concepts.

Shift operation means that we should not link the current two concepts directly because they will share a common ancestor concept that should be found.

For two adjacent concepts c_m and c_n, if both the DirAdjStr(c_m, c_n) and the IndAdjStr(c_m, c_n) are less than a threshold, their operation will be abandon. Otherwise, we need to decide which operation should be chosen, and their relations also should be further assigned. To do this, our basic ideas are as follows.

First, the similarity between two concepts’ attribution collections and their degree ratio can indicate their relation types. For example, if two concepts’ attributions are highly similar, they will probably have an equivalence relation. With the help of some linguists, we demonstrate our understanding of the different relations in Figure 6. In this figure, each cycle represents a concept, each undirected edge represents a kind of attribution collection and each directed edge represents a kind of relation.

Figure 6.

Experts’ understanding for concept relations.

Second, if two adjacent concepts are in different periods of the life cycles, their similarity should be adjusted. The basic idea for this adjustment is that, if two concepts are in different periods, the similarity between them should be discounted. Moreover, the greater two concept periods’ distance is, the larger the discount ratio that should be assigned to their similarity.

Based on these ideas, we designed a relation assignment algorithm shown in Figure 7. In fact, this algorithm is composed of some heuristic relation decision rules that are transformed from the information shown in Figure 6. These heuristic rules are designed with the help of some human experts. Therefore, Figures 6 and 7 can be seen as the same information shown in different ways. The similarities in Figure 7 are computed by the formulas (5)–(8).

Figure 7.

Algorithm for relation assignment in heuristic learning method.

DegSim (c_{m}, c_{n}) = \frac{1}{1 + pdist (c_{m}, c_{n})} \times \frac{deg (c_{m})}{deg (c_{n})}

(5)

AttrSim (c_{m}, c_{n}) = \frac{1}{1 + pdist (c_{m}, c_{n})} \times (φ_{1} \sum_{p \in Cyc (c_{m}) \cap^{C} yc (c_{n})} \sum_{i = 1}^{i = 5} α_{i} AccuSi m_{p} (tupl e_{i} (c_{m}), tupl e_{i} (c_{n})) + φ_{2} \sum_{i = 1}^{i = 5} α_{i} AccuSi m_{Cyc} (tupl e_{i} (c_{m}), tupl e_{i} (c_{n})) + φ_{3} (deg (c_{m}) - deg (c_{n})))

(6)

AccuSim (tupl e_{i} (c_{m}), tupl e_{i} (c_{n})) = \frac{2 \times ϑ_{1} | tupl e_{i} (c_{m}) \cap tupl e_{i} (c_{n}) |}{| tupl e_{i} (c_{m}) | + | tupl e_{i} (c_{n}) |} + \frac{ϑ_{21} | tupl e_{i} (c_{m}) |}{| tupl e_{i} (c_{n}) |}

(7)

StrSim (c_{m}, c_{n}) = \frac{2 \times ComStr (c_{m}, c_{n})}{len (c_{m}) + len (c_{n})}

(8)

In above formulas, 1/(1 +pdist(c_m, c_n)) is the discount ratio used for adjusting the similarity between two concepts and pdist(c_m, c_n) means the period distance between c_m and c_n. Here we suppose that the four life cycle periods are distributed in a one-dimensional space and their corresponding coordinates can be viewed as a ratio of the number of displayed attributions for a concept to the number of its total owned attributions. Thus the parameter pdist(c_m, c_n) can be defined as the absolute value of the period coordinates’ subtraction between c_m and c_n. Moreover, stability period and extinction period can be assigned the same coordinates because we think that a concept’s attributions have been fully displayed in either of them. Specifically, in our latter experiments, the coordinates of life cycle periods are empirically set as 0, 0.5, 1 and 1 respectively. In formulas (6) and (7), the parameter tuple_i(c_m) means the ith attribution tuple for concept c_m. (See previous definitions described in Section 3.1, where a concept’s attribution is defined as a five-tuple.) In formula (8), ComStr(c_m, c_n) is the common character in c_m and c_n. There are several parameters that should be set in the above formulas.

In Figure 6, when the operation type is shift, we need to find a common ancestor concept c_k for the input concepts c_m and c_n. Here we think c_k should be a concept that satisfies the following condition:

c_{k} = argma x_{k} (DegSim (c_{k}, c_{m}) \times DegSim (c_{k}, c_{n}) + AttrSim (c_{k}, c_{m}) \times AttrSim (c_{k}, c_{n}))

(9)

When c_k is found, the same algorithm as described in Figure 7 is used to assign the proper relations among c_k, c_m and c_n.

3.2.6. Classification-based learning method

In this classification-based learning method, we regard the relation assignment task as a classification task. All of the factors discussed previously can be taken as features to train a classification model. Specifically, the features used for every concept pair are listed in Table 2.

Table 2.

Used features for a concept pair (c_i, c_j).

Features	Descriptions
deg(c_i), deg(c_j)	Concept degrees
per(c_i), per(c_j)	The life cycle periods where two concepts are in
fyer(c_i), fyer(c_j)	The first appearances for two concepts
lyer(c_i), lyer(c_j)	The duration years for two concepts
AdjStr(c_i, c_j)	The adjacent strength for two concepts
DirAdjStr(c_i, c_j)	The directly adjacent strength for two concepts
IndAdjStr(c_i, c_j)	The indirectly adjacent strength for two concepts
AccuAttr(c_i, t_m,n)	c_i time accumulation attribution in all past years
AccuAttr(c_i, t_m,n)	c_j time accumulation attribution in all past years
AttrVar(c_i, t_m,n)	c_i attribution variation rate averaged in all past years
AttrVar(c_j, t_m,n)	c_j attribution variation rate averaged in all past years
StrSim(c_i, c_j)	Two concepts’ string similarity computed with formula (8)

During the process of training a classification model, every two adjacent concepts are taken as a sample which is to be assigned a relation type. A possible relation set is {hierarchical, equivalence, has-a, part-of, kind-of }.

3.2.7. Subset selection strategy

There are two subsets that should be carefully selected according to our relation learning method, as shown in Figure 5. One is for labelling, and the other is for re-labelling. Here we use a simple method for these two subset selection tasks.

For the first one, we use a random selection strategy. That is to say, we will randomly select n unlabelled concept pairs each time for the trained classification model to label their relation types. This is easy to understand: since we do not know which concept pair is more important compared with others, we just select them randomly.

For the second one, we use a classification probability-based selection strategy. When using the trained classification model to label the relation type of each concept pair, there will be a probability along with the relation type. This probability is used to denote the system’s confidence for current type output to some extent. The lower the probability is, the more possible it is that current output would be wrong. Thus here we can select the lowest-k results for re-labelling. This is easy to understand: if we can tell the correct results for those outputs for which the classification model is not confident, we can expect that the retrained model will be better.

3.3. Parameter weight tuning

There are some parameter weights that need to be tuned in both the process of concept life cycle division and the process of relation learning. For the first process, only two parameters’ weights needs to be tuned δ and γ, which are used in Table 1. For them, a simple 10-fold cross-validation method is used to select the best parameter values. For the second process, there are some parameters’ weights that need to be tuned for every kind of relation. We design a perceptron-based weight tuning method as shown in Figure 8. In our method, every kind of relation will have a parameter set. During the parameter tuning process, every concept pair can be expressed as a parameter vector according to Figure 7. Thus in a training sample (x_i, y_i), x_i is a concept pair’s parameter vector and y_i∈ {+1, −1}, which denotes a positive sample or a negative sample, respectively. The parameter weight vector is denoted w. In Figure 8, the learning rate η is a step parameter and here we set it to 0.01.

Figure 8.

Weight tuning algorithm for heuristic relation learning method.

3.4. Result demonstration

Taking ‘machine translation’ as an example, Figure 9 demonstrates a small part of its related concepts along with their relations (in fact, there are more related concepts and relations in the real obtained domain ontology).

Figure 9.

An example of the constructed domain ontology.

The final obtained domain ontology consists of a tuple set where each item has such a format like ‘concept₁, concept₂, RelVector’. In such a tuple, the RelVector component is a five-dimensional vector that corresponds to our used relation set {hierarchical, equivalence, has-a, part-of, kind-of}. Each dimension has a 1/0 binary value, which indicates concept₁ has/has not a corresponding relation with concept₂. In Figure 9, the solid arrows are used to indicate hierarchical relations, while the dotted arrows are used to indicate other kinds of relations. Besides, according to previous relation definitions, has-a relation is a reverted relation of part-of relation, so in Figure 9 we only show the part-of relations for simplicity and clarity. With such an ontology graph, one can easily see that ‘C-E machine translation’ is a kind-of ‘machine translation’, ‘sentence similarity’ is a part-of ‘example based machine translation’ or ‘example based machine translation’ has-a ‘sentence similarity’, and so on.

4. Experimental evaluation

4.1. Basic settings

4.1.1. Dataset

We used Chinese academic papers as our dataset because there is a very easy way to obtain them: almost all of Chinese published academic papers can be downloaded from a website (https://www-cnki-net.web.bisu.edu.cn/) and all of them have been classified into the proper domains. From this website, we downloaded more than 400,000 academic papers in the Information Technology domain, spanning the past 30 years (1979–2010).

4.1.2. Dev/test set

We manually constructed two dev sets, each of which contained 1000 samples: one was used for parameter tuning and the other was used for classification model training. In addition, we also manually constructed a test set containing 500 samples. For all of the concept pairs in these samples, all of their possible relations were carefully assigned by some human experts. That is to say that there may be more than one kind of relation for some concept pairs. Brief statistics about these dev/test sets are shown in Table 3.

Table 3.

Brief statistics on the dev/test set.

	Number of samples
	Dev set 1	Dev set 2	Test set
equivalence	9	11	4
has-a	478	486	197
part-of	478	486	197
kind-of	302	283	139
hierarchical	612	598	257

4.1.3. Baseline system

We take the method described in Ren [1, 2] as the baseline. The reasons for this selection are as follows. First, this method also aims to construct a Chinese domain ontology. Second, this method uses the same data to construct a domain ontology. For the baseline method, we use the same parameter setting as Ren [1, 2]. However, it is worth noting that, in the baseline method, they only focused on the hierarchical concept relation, that is to say, they only output hierarchical relations in their methods. We did not select those pattern based concept relation learning methods as the baseline system for the following reasons. In our method, we take the keywords as domain concepts directly, and there is no suitable content information for them. Therefore, we think that it would be unfair to compare our method with those pattern-based methods.

4.1.4. Evaluation strategy

Generally there are two main methods that are often used to evaluate a given ontology. One is to evaluate the quality of a given ontology directly by the metrics such as accuracy and recall. The other applies the target ontology to a specific application, and evaluates the quality of target ontology by comparing the performance of the specific application before and after using this ontology. In this paper, we use the first evaluation method because (1) it can give a direct evaluation for a given target ontology and (2) the reliability of the second method depends heavily on the selected application. Specifically, precision, recall and F₁ score are used for the evaluation.

4.2. Performance evaluation

We first conducted an experiment to evaluate the effectiveness of our proposed method. In the classification-based relation learning method, the following classification models were used. During the subset selection phase, the parameter n for labelling selection was set as 200,000, and the parameter k for re-labelling selection was set as 300. The termination condition in our concept relation learning method was iteration times which were set at 10. (It is worth noting here that the final number of training samples will be proportional to the iteration times, so the iteration times cannot be too large for the sake of classification efficiency and classification precision.)

SVM [44] is a supervised learning method. In our experiment, we trained an SVM model for each kind of relation, and the 1-verse-other strategy is used when training the classification model. We used an open source package LIBSVM³ and the linear kernel function.

Naive Bayesian (NB) is a simple probabilistic classifier by applying Bayesian theorem with strong independence assumptions. We used a multi-nomial naive Bayesian classifier in our experiment [45].

Maximum entropy (ME) is also a probabilistic classifier model by applying the maximum entropy theory. We used an open source package⁴ in our experiment.

Tables 4 and 5 display the performance in terms of different metrics. (HLM refers to the Heuristic Learning Method introduced in Section 3.2.3. Here we list the performance of the heuristic learning method to give readers more information about the learning ability of those rule-based learning methods.)

Table 4.

Performance comparisons with different methods.

	Relation types	Precision	Recall	F ₁
	hierarchical	83.39%	89.88%	86.51%
	equivalence	20.0%	50.0%	28.57%
HLM	has-a	68.61%	77.67%	72.86%
	part-of	68.61%	77.67%	72.86%
	kind-of	62.96%	61.15%	62.04%
	hierarchical	89.67%	94.55%	92.05%
	equivalence	25%	50%	33.33%
HLM + SVM	has-a	85.78%	91.88%	88.73%
	part-of	85.78%	91.88%	88.73%
	kind-of	62.18%	69.78%	65.76%
	hierarchical	83.15%	88.33%	85.66%
	equivalence	27.27%	75%	40%
HLM + NB	has-a	83.41%	91.88%	87.44%
	part-of	83.41%	91.88%	87.44%
	kind-of	62.94%	64.75%	63.83%
	hierarchical	85.34%	88.33%	86.81%
	equivalence	28.57%	50.00%	36.36%
HLM + ME	has-a	86.70%	89.34%	88.0%
	part-of	86.70%	89.34%	88.0%
	kind-of	64.96%	54.68%	59.34%
Baseline	hierarchical	75.1%	78.2%	76.6%

Table 5.

Performance comparisons about our used method.

	microP	microR	microF
HLM	71.89%	78.59%	75.09%
HLM + SVM	82.15%	88.67%	85.28%
HLM + NB	79.21%	85.89%	82.42%
HLM + ME	82.54%	82.74%	82.64%

It is observed that both the hierarchical learning method and our hybrid learning method achieve better performance compared with the baseline method. We think the reason is that our method takes more factors into consideration compared with the baseline method. In addition, it can also be observed that the hybrid method achieves better performance compared with the heuristic learning method. We think this is mainly for the following reason. The heuristic learning method uses many human experts’ experiences, but the contributions of these experiences are difficult to assessed properly. On the contrary, with a classification-based learning phase, the contributions of different factors are determined automatically by the models themselves. Thus the final classification model is better to train better learning models.

Of course, it can also be observed from Table 4 that our method still has a large improvement space for some kinds of relations, such as equivalence relation and kind-of relation. Both of them obtain lower F₁ scores compared with other relations. We think these poor performances mainly stem from the following reason: current considered factors cannot provide enough clues for deciding these relation types. For example, it is hard to figure out that ‘naive bayesian’ is a kind of ‘classifier’ by our current methods. We think deeper analysis technologies should be used.

4.3. Robustness evaluation on larger datasets

To evaluate our learning method’s robustness on larger dataset, we conducted other experiments to evaluate the performance of our method on different data size. To do this, we first used the trained classification models to construct domain ontologies on different sized of dataset. Then we randomly selected 500 samples from these result sets as test data to evaluate their qualities. For evaluation simplicity, only the microprecision was used as an evaluation metric here. The evaluation results are shown in Table 6.

Table 6.

Different methods’ performance on different data sizes.

Data size	microP
	HLM	HLM + SVM	HLM + NB	HLM + ME
1000	72.2%	82.2%	71.6%	81.2%
2000	73.4%	83.8%	73.4%	84%
5000	71%	79.4%	70.4%	78.4%
10,000	69.8%	73.4%	65.8%	72.4%
30,000	67.8%	70.2%	65%	69.2%
50,000	68.2%	67.6%	62.6%	65.8%
100,000	55.4%	64.6%	60.4%	61.4%
200,000	56.2%	64.2%	56.2%	57.8%
500,000	51.2%	60.4%	54.2%	56.2%
The whole dataset	41.2%	57.8%	50.8%	50.4%

From Table 6 we can see that the performance of all methods decreases when the data size increases. However, the hybrid method obtains better results on the larger dataset. We think that this is because of the following reasons. First, in the Heuristic Learning Method, human experts’ experiences play a more important role in the relation type decision. Such experiences would work well on those visible samples. However, once there are some decision rules that are not covered by current rules, their learning performances will decrease greatly. Second, in the Heuristic Learning Method, its decision rules are mainly based on frequency information and co-occurrence information; these kinds of information will not be so proper to process some complicated cases, especially as often occurs when the sample scale increases to a very large size. Of course, the hybrid method’s performance also decreases with the data size increase. We think this is because, when the data size increases, the ratio of training set to test set will be less and less. Thus the classification learning model is more prone to lead to an overfitting problem with smaller training set.

In addition, as mentioned before, the performances for different types of relations differ greatly. Table 7 shows this difference when we use a dataset that contains 500,000 samples. In this experiment, we also manually label all the relation types for each test concept pair.

Table 7.

Performance comparisons with different methods with 500,000 samples.

	Relation types	Precision	Recall	F ₁
	hierarchical	88.35%	63.17%	73.67%
	equivalence	40%	28.57%	33.33%
HLM	has-a	59.55%	78.33%	67.66%
	part-of	59.55%	78.33%	67.66%
	kind-of	69.23%	51.59%	59.12%
	hierarchical	81.55%	59.41%	68.74%
	equivalence	25%	28.57%	26.67%
HLM + SVM	has-a	43.71%	65.02%	52.28%
	part-of	43.71%	65.02%	52.28%
	kind-of	64.54%	57.96%	61.07%
	hierarchical	75.45%	56.18%	64.41%
	equivalence	18.18%	28.57%	22.22%
HLM + NB	has-a	42.86%	60.59%	50.20%
	part-of	42.86%	60.59%	50.20%
	kind-of	50.64%	75.80%	60.71%
	hierarchical	76.56%	56.18%	64.81%
	equivalence	23.08%	42.86%	30%
HLM + ME	has-a	41.72%	69.46%	52.13%
	part-of	41.72%	69.46%	52.13%
	kind-of	46.67%	62.42%	53.41%
Baseline	hierarchical	67.25%	71.77%	69.44%

5. Discussion

In this paper, we focused our method on the application scenario of scientific papers. In fact, the proposed method is not confined to scientific papers. On the contrary, it can be widely used in other scenarios, for example, market surveys, product reviews, patent prior search, and so on. What is different is that, in these application scenarios, there may be no available keywords that can be directly used as the theme keywords. Thus an extra theme keyword extraction procedure should be well designed.

In addition, our method is also language independent. Neither deep linguistic knowledge nor domain- or language-specific annotated corpora is required. That is to say one can easily transplant the proposed method to similar application scenarios in other languages.

However, it is worth noting that the performance of the method will rely on the human expert’s experience. Thus it would work well on those test examples that are similar in attribution distributions to those examples used for classification model training. When some new unseen example emerges, the method may not work so well. In fact, this is also an open issue for most human-assisted learning methods. However, this issue would be alleviated greatly by our subset selection strategies described in Section 3.2.5. As Figure 5 shows, our relation learning process is an iteration process, and in each loop, some learning results that have lower confidences (the confidences are decided by classification probability output by classification system) are picked out for re-labelling by human experts again. These lower confidence examples are more likely to be some unseen examples in the previous loop. Therefore the global learning performance will increase greatly after several learning loops.

6. Conclusions

In this paper, we propose a hybrid learning method that aims to learn domain ontology from Chinese scientific papers. The main contributions of this research can be summarized as follows. First, to the best of our knowledge, this work is the first attempt towards taking the time factor into consideration during ontology construction. Second, our approach is a hybrid method that combines human expert knowledge and a classification-based learning scheme, and uses both local and global information.

Footnotes

Funding

This paper is supported by the ‘Fundamental Research Funds for the Central Universities’ and the National Natural Science Foundation of China (grant nos 61003159, 61100089, 61073140 and 61272376).

Notes

References

Ren

. A demo for constructing domain ontology from academic papers. In: Proceedings of COLING 2012: demonstration papers, 2012, pp. 369–376.

Ren

. A cheap domain ontology construction method based on graph generation and conversion method. Journal of Information and Computational Science 2012; 9(18): 5823–5830.

Brijesh

Pushpark

. Domain specific ontology extractor for Indian languages. In: Proceeding of the 10th workshop on Asian language resources, 2012, pp. 75–84.

Buitelaar

Olejnik

Sintek

. A Protégé plug-in for ontology extraction from text based on linguistic analysis. In: Proceedings of the 1st European semantic web symposium, 2004, pp. 31–44.

Boonchom

Soonthornphisaj

. ATOB algorithm: An automatic ontology construction for Thai legal sentences retrieval. Journal of Information Science 2012; 38(1): 37–51.

Navigli

Velardi

Gangemi

. Ontology learning and its application to automated terminology translation. IEEE Intelligent Systems 2003; 18(1): 22–31.

Navigli

Velardi

. Learning domain ontologies from document warehouses and dedicated web sites. Journal of Computational Linguistics 2004; 30(2): 151–179.

Cimiano

Volker

. Text2Onto – A framework for ontology learning and data-driven change discovery. In: Proceedings of the 10th international conference on application of natural language to information systems, 2005, pp. 227–238.

Ruiz-Martínez

Valencia-García

Fernández-Breis

García-Sánchez

Martínez-Béjar

. Ontology learning from biomedical natural language documents using UMLS. Expert Systems with Applications 2011; 38(10): 12365–12378.

10.

Valencia-García

Fernández-Breis

Ruiz-Martínez

García-Sánchez

Martínez-Béjar

. A knowledge acquisition methodology to ontology construction for information retrieval from medical documents. Expert Systems: The Knowledge Engineering Journal 2008; 25(3): 314–334.

11.

Ochoa

Garcia

Soltero

Valenzuela

. A semantic role labelling-based framework for learning ontologies from Spanish documents. Expert Systems with Applications 2013; 40: 2058–2068.

12.

Sabou

Wroe

Goble

Struckenschmidt

. Learning domain ontologies for semantic Web service descriptions. Web Semantics: Science, Services and Agents on the World Wide Web 2005; 3: 340–365.

13.

Shamsfard

Barforoush

. Learning ontologies from natural language texts. International Journal of Human–Computer Studies 2004; 60(1): 17–63.

14.

Sánchez

Moreno

. Learning non-taxonomic relationships from web documents for domain ontology construction. Data and Knowledge Engineering 2008; 64; 600–623.

15.

Dahab

Hesham

Hassan

. TextOntoEx: Automatic ontology construction from natural English text. Expert Systems with Applications 2008; 34(2): 1474–1480.

16.

Villaverde

Persson

Godoy

Amandi

. Supporting the discovery and labeling of non-taxonomic relationships in ontology learning. Expert Systems and Applications 2009; 36(7): 10288–10294.

17.

Hou

Ong

Nee

AYC

Zhang

Liu

. GRAONTO: A graph based approach for automatic construction of domain ontology. Expert Systems with Applications 2011; 38(9): 11958–11975.

18.

Zouaq

Gasevic

Hatala

. Towards open ontology learning and filtering. Information Systems 2011; 36(7): 1064–1081.

19.

Sánchez

Moreno

Vasto-Terrientes

. Learning relation axioms from text: An automatic Web-based approach. Expert Systems with Applications 2012; 39(5): 5792–5805.

20.

Lee

Kao

Kuo

Wang

. Automated ontology construction for unstructured text documents. Data & Knowledge Engineering 2007; 60: 547–566.

21.

Zou

Park

. Modeling ancient Chinese time ontology. Journal of Information Science 2011; 37(3): 332–341.

22.

Jiang

Tan

. CRCTOL: A semantic-based domain ontology learning system. Journal of the American Society for Information Science and Technology 2010; 61(1): 150–168.

23.

Kang

Wang

. Research on domain ontology in different granulations based on concept lattice. Knowledge-Based Systems 2012; 27: 152–161.

24.

Chen

Liang

Pan

. Using recursive ART network to construction domain ontology based on term frequency and inverse document frequency. Expert Systems with Applications 2008; 34: 488–501.

25.

Shih

Chen

Chu

Chen

. Enhancement of domain ontology construction using a crystallizing approach. Expert Systems with Applications 2011; 38: 7544–7557.

26.

Hsieh

Lin

Chi

Chou

Lin

. Enabling the development of base domain ontology through extraction of knowledge from engineering domain handbooks. Advanced Engineering Informatics 2011; 25: 288–296.

27.

Wei

Wang

. From web resources to agricultural ontology: A method for semi-automatic construction. Journal of Integrative Agriculture 2012; 11(5): 775–783.

28.

Meng

Wang

. Large-scale ontology development and agricultural application based on knowledge domain framework. Journal of Integrative Agriculture 2012; 11(5): 808–822.

29.

Zhang

. An approach to automatically constructing domain ontology. In: Proceedings of the 20th Pacific Asia conference on language, information and computation, 2006, pp. 150–157.

30.

Brusa

Caliuscro

Chiotti

. Towards ontological engineering: A process for building a domain ontology from scratch in public administration. Expert Systems 2008; 25(5): 484–503.

31.

Lumsden

Hall

Cruickshank

. Ontology definition and construction, and epistemological adequacy for systems interoperability: A practitioner analysis. Journal of Information Science 2011; 37(3): 246–253.

32.

Tian

Ren

. Learning relation instances for Chinese domain ontology from the web. IEEJ Transactions on Electrical and Electronic Engineering 2010; 5(2): 188–194.

33.

Khoo

CSG

. Semantic relations in Information science. Annual Review of Information Science and Technology 2006; 40(1): 157–228.

34.

Legg

. Ontologies on the Semantic Web. Annual Review of Information Science and Technology 2007; 41(1): 407–451.

35.

Tan

Lambrix

. Selecting an ontology for biomedical text mining. In: Proceedings of the Workshop on BioNLP, 2009, pp. 55–62.

36.

Estival

Nowak

Zschorn

. Towards ontology-based natural language processing. In: Proceedings of the workshop on NLP and XML, 2004, pp. 59–66.

37.

Kietz

Volz

. Extracting a domain-specific ontology from a corporate intranet. In: Proceedings of the 2nd workshop on learning language in logic and the 4th conference on computation natural language learning, 2000, pp. 167–175.

38.

Maedche

Staab

. Ontology learning for the Semantic Web. IEEE Intelligent Systems 2001; 16(2): 72–79.

39.

Salem

AbdeRahman

. A multiple-domain ontology builder. In: Proceedings of the 23rd international conference on computation linguistics, 2010, pp. 967–975.

40.

Trapman

Monachesi

. Ontology engineering and knowledge extraction for crosslingual retrieval. In: Proceedings of the international conference of recent advances in natural language processing, 2009, pp. 455–459.

41.

Vela

Declerck

. Concept and relation extraction in the finance domain. In: Proceedings of the 8th international conference on computational semantics, 2009, pp. 346–350.

42.

Huang

Yang

Chen

. An ontology of Chinese radicals: Concept derivation and knowledge representation based on the semantic symbols of four hoofed-mammals. In: Proceedings of the 22nd Pacific Asia conference on language, information and computation, 2008, pp. 189–196.

43.

Huang

Sinica

. Text-based construction and comparison of domain ontology: A study based on classical poetry. In: Proceedings of the 18th Pacific Asia conference on language, information and computation, 2004, pp. 17–20.

44.

Cortes

Vapnik

. Support-vector networks. Machine Learning 1995; 20(3): 273–297.

45.

Yang

Pederson

. Feature selection in statistical learning of text categorization. In: Proceedings of international conference on machine learning, 1997, pp. 412–420.