An improving SRL model with word sense information using an improved synergetic neural network model

Abstract

As a fundamental task of natural language processing, semantic role labeling (SRL) have attracted much attention of researchers in recent years. However, with increasing features being added into the studies, the performance growth trend of SRL is gradually slowing down. So new ways must be found to improve the performance of semantic analysis. Word sense information is useful for SRL task. But how to effectively make use of word sense information is a key issue. Referring to synergetics, we can regard semantic analysis process as competitive process of many semantics order parameters under coherent action and interactive collaboration of semantic role-related features and word sense-related features. Accordingly, we propose a semantic role labeling model with word sense information based on improved synergetic neural network (SNN). Our contributions are three-fold. Firstly, role-related features and word sense-related features are used to configure semantic order parameters of SNN. Secondly, network parameters are reconstructed which can reflect the relationship of driving and restraining each other between various linguistic features. Finally, we use an improved quantum particle swarm algorithm (QPSO) to realize the optimization of network parameter which has stronger search ability and faster convergence speed. By evaluating our model on the OntoNotes 2.0 corpus, the experiment results show the proposed model in this paper leads to a higher performance for SRL.

Keywords

Word sense SRL SNN Semantic order parameters QPSO

1 Introduction

Automatic semantic analysis is an important task in natural language processing research. However, limited to the current level of technology, it is more difficult to make deep semantic analysis. Semantic role labeling [1, 2] is a simplified method of semantic analysis, which does not make complete semantic analysis for the entire sentence, but only labels predicate-related semantic role. Semantic role labeling enjoys many advantages such as clear definition of issues, easiness for manual labeling, etc, and thus has broad application prospects. A good processing of semantic role labeling issue will help solve other natural language processing tasks such as machine translation [3], question answering [4, 5] and information extraction [6].

At present, the mainstream studies of semantic analysis employ a variety of statistical machine learning techniques and linguistics features for semantic recognition. However, with more and more features added, improvement trend of semantic analysis performance gradually slows down. Some scholars adopt detailed and rich feature engineering to improve the performance of SRL system. These features are primarily derived from lexical information, part of speech information and syntactic information. But for labeling models, features extracted from the lexical information of words are too sparse, while features extracted from part of speech are too generalized. Word sense-related information may reflect indescribable relationship between lexical information and part of speech information. Introduction of word sense will help to determine semantic roles to further improve the performance of labeling system. But currently there are still few studies which apply word sense information in semantic role labeling. Only a small part of scholars made some attempts. Surdeanu et al. [7] first proposed a predicate recognition model in CONLL 2008 and this task can be regarded as word sense disambiguation. Meza-Ruiz and Riedel [8] attempted to improve performance of semantic role labeling with predicate word sense information. Che et al. [9] considered application of whole word sense information in semantic role labeling system, and discussed impact of predicate-related features and word-related features on systemperformance.

In semantic role labeling process, the introduction of word sense information is expected to achieve better results. But how to better reflect the relationship between various semantic features is a key issue. In current mainstream SRL systems (SVM-based model [10]; MaxEnt-based model [11, 12]; Decision tree model [13]; AdaBoost [14]), semantic role information and word sense information are considered to be independent, possible dependencies between them are ignored and full use of semantic information is incapable.

For a better mutual learning, this paper introduces synergetic pattern recognition [15] into semantic analysis, which can better represent global constraint of semantic role information and word sense information, so that semantic role labeling can achieve global optimal solution in the integration process. Compared with other pattern recognition methods, synergetic neural network enjoys greater advantages in dealing with fuzzy matching applications, such as face recognition [16], image retrieval [17] and mangroves classification [18]. Problem of incomplete contextual information often occurs in semantic annotation process, such as information missing in part of speech, word sense information and semantic role information, etc. Application of synergetic neural network in processing of semantic annotation problem is expected to achieve better performance. An improved SNN-based SRL model is proposed and implemented in this paper which can better handle fuzzy matching of semantic information, so as to improve the performance of SRL.

This paper is organized as follows. In Section 2, some related work is introduced. A brief introduction to SNN and SNN-based SRL model is described in Section 3. An improved SRL model with word sense is introduced in section 4. Some experimental results and conclusions are given in the last section.

2 Related work

In general, semantic role labeling and word sense disambiguation are considered to be independent. Word sense information is rarely used in semantic role labeling. In recent years, A few researchers have carried out a small number of related research. Dang and Palmer [19] used semantic role information in verb sense disambiguation, experiment showed the performance of word sense disambiguation can be improved by leveraging information about predicate arguments. Hajič et al. [20] used the predicate sense information to help semantic role labeling. Dahlmeier et al. [21] reported a joint model for word sense disambiguation of prepositions and semantic role labeling of prepositional phrases, which can lead to an improvement for both subtasks. Che and Liu [22] propose a joint model of semantic role labeling and word sense disambiguation based on markov logic model, which can further improve the performance of semantic analysis.

Synergetic neural network [15] is a pattern recognition method proposed by German physicist professor Haken which extends synergetic theory into cognitive science and computer science by taking advantage of deep similarities between spontaneous pattern formation and pattern recognition based on synergetics. Synergetic neural network is without pseudo-state, but with relatively strong anti-noise and anti-defect capacity, and its order parameter evolutionary process conforms to human cognitive processes. In recent years, extensive research has been carried out in learning algorithm of SNN, especially in the setting of attention parameter [23 –25], the selection of prototype pattern vector [26], and reconstruction algorithm of order parameters [27] and so on.

In the learning process of synergetic neural network, attention parameters can play an important role in biased regulation. By adjusting synergetic attention parameters, self-learning ability of synergetic neural network can be fully utilized to improve recognition results of the network. Fang et al. [23] adopt award-penalty learning method for attention parameters, which is to accordingly increase attention parameters that need to be correctly classified based on false identification of test pattern in the learning process, so as to decrease attention parameters of misidentified class. This dynamic adjustment of attention parameters enables the system to achieve best recognition performance. Ma et al. [24] believes that mediation of synergetic parameter is a global behavior, which introduces immune clone algorithm into optimization of synergetic attention parameter, and obtains better recognition performance by setting mutation probability and cloning scale. However, this method features increased search space and slow convergence rate. Zou et al. [25] proposed a parameter optimization algorithm based on differential evolution algorithm, which improves global and local optimization ability of algorithm. Nevertheless, the method is with slow convergence rate, not desirable in some practical applications.

In this paper, an improved quantum particle swarm optimization [27, 28] is proposed to implement the parameter optimization of synergetic neural network which can avoid some shortcomings of the traditional evolutionary algorithm, such as poor global convergence ability and slow convergence rate.

3 An integrative semantic analysis method based on synergetic neural network

3.1 A brief introduction to SNN

Synergetics mainly studies how each individual in complex system makes qualitative change of system state through mutual synergies. The basic idea is: The cooperative system is composed of various subsystems, and when each subsystem is far from equilibrium, synergies and coherence effects will be generated, a certain critical point will then be achieved, so that the system state has fundamental change in time and space, changing from disordered state into ordered state. Professor Haken pointed out that new models and new features formed by self-organization are governed by the sameprinciples.

A certain number of order parameters can be constructed according to test pattern q and prototype pattern, evolve with kinetic equation, and eventually drive test pattern q to enter a prototype pattern v_k from the intermediate state q (t), thus completing identification of q, as shown in formula (1). $q (0) \to q (t) \to v_{k}$ (1)

Haken noted that synergetic pattern recognition process can be described with potential kinetic equation. If fluctuation force F (t) and transient component are ignored, M is the number of prototype pattern vector, then potential function equation is:

$\begin{matrix} V & = & - \frac{1}{2} \sum_{k = 1}^{M} λ_{k} {(v_{k}^{+} q)}^{2} + \frac{1}{4} B \sum_{k^{'} \neq k} {(v_{k^{'}}^{+} q)}^{2} \\ + \frac{1}{4} C {(\sum_{k = 1}^{M} {(v_{k}^{+} q)}^{2})}^{2} \end{matrix}$ (2)

Corresponding kinetic equation is: $\dot{q} = - \frac{\partial V}{\partial q^{+}}$ (3)

Substitute order parameter $ξ_{k} = v_{k}^{+} q$ into the formula (2), and then corresponding kinetic Equation of order parameter (4) and potential Equation (5) can be obtained. ${\dot{ξ}}_{k} = λ_{k} ξ_{k} - B \sum_{k^{'} \neq k} ξ_{k^{'}}^{2} ξ_{k} - C (\sum_{k^{'} = 1}^{M} ξ_{k^{'}}^{2}) ξ_{k}$ (4) $\begin{matrix} V & = & - \frac{1}{2} \sum_{k = 1}^{M} λ_{k} ξ_{k}^{2} + \frac{1}{4} B \sum_{k^{'} \neq k} ξ_{k^{'}}^{2} ξ_{k}^{2} \\ + \frac{1}{4} C {(\sum_{k^{'} = 1}^{M} ξ_{k^{'}}^{2})}^{2} \end{matrix}$ (5)

According to the above synergetic theory, kinetic equation of order parameter can be discretized into formula (6). $ξ_{k} (n + 1) = ξ_{k} (n) + r (λ_{k} - D + B ξ_{k}^{2} (n)) ξ_{k} (n)$ (6)

In the formula $D = (B + C) \sum_{k^{'}} ξ_{k^{'}}^{2} (n)$ , n represents the current iteration number, r is the step size. λ_k is attention parameter, while B and C are given parameters.

3.2 A SNN-based semantic role labeling model

This section constructs SRL model based on synergistic interaction between semantic roles, as shown in Fig. 1, which mainly contains order parameter configuration network and order parameter evolving network. In order parameter configuration network, configuration of semantic role chain order parameter is achieved. In order parameter evolving network, evolution equation configuration and network parameter configuration are implemented. Through iterative competition of semantic order parameters in dynamics equation, marked role chain is ultimately obtained.

The order parameter configuration network should complete the following tasks:

Obtain feature vector from training corpus, and construct prototype pattern vectors;

Obtain feature vector from test corpus, and construct test pattern vectors;

Configure order parameter that reflects similarity between prototype pattern vector and test pattern vector;

By sorting order parameter, obtain N-best selected roles;

Assembly semantic roles to obtain candidate roles chain, and calculate order parameter of roles chain.

In order parameter evolving network, the following tasks must be completed:

Configure dynamic evolution equation;

Set network parameter;

Through evolution of order parameter equation, obtain the final labeling pattern.

Semantic role labeling process can be seen as a competitive process of many roles chain order parameters, and order parameter with the largest support will win, thereby obtaining the final labeling results.

4 Semantic role labeling based on improved SNN with word sense

How to make full use of word sense information is a problem to be solved in the SNN-based SRL systems. Order parameter conversion is to make initial order parameter of corresponding pattern have a dominant position in the competition, and the role of regulation parameters can make the pattern whose initial order parameter is not the biggest win in the competition; so how to coordinate configuration between them in the optimization process is worthy of further study.

In this section, we implement a synergetic semantic role labeling model with the introduction of word sense information, as shown in Fig. 2. First, some role-related feature and word sense-related feature are added to semantic vector space model. Second, network parameters are reconstructed according to logical constraints that exist in semantic roles chain, to better reflect the logical relationship between word senses and semantic roles. Finally, the best roles chain will be obtained by the evolution of semantic order parameters.

4.1 Feature engineering

The features used in our model contain some basic features and extended features. These features have been proved to be effective for SRL task [30, 31]. To better reflect the role of word sense information, we also select a number of word sense-related features which are introduced in some SRL system [22]. This feature includes:

Basic features: Predicate lemma, Predicate part of speech (POS), Predicate voice, Dependency, Predicate Path (The dependency chain from the current node to the current predicate), The dependency chain of predicate child, Predicate subclass framework, Head word and Head word path (the sequence of headword node relative to the current predicate).

Expansion features: Syntactic path, Syntactic path length, Partial syntactic path length, Relation path, Partial relation path, Partial syntactic path, Dependency chain of predicate brothers, Predicate relation, Predicate+syntactic path, Syntactic subclasses framework of predicate, Head word chain of predicate brothers, Predicate+head word, POS of head word, Head word+POS and Current relation of head word.

Word sense-related features

Lemma+Sense: For example sheep.n.1 (n represents that it is a noun, 1 is meaning number corresponding to WordNet).

Hypersense(n): The n-layer hypernym sense.For example, hypernym sense of “Sheep.n.1” is “bovine.n.1”, wherein, variable n denotes the n-layer hypernym sense.

For example, in WordNet, it can be found that hypernym sense corresponding to sheep.n.1 is bovine.n.1. Root sense is eventually obtained by continuous search. Through hypernym sense bovine.n.1 of sheep.n.1, it can be found that its hypernym sense is ruminant.n.1. If this process continues, root sense entity.n.1 can be eventually found. When n = 1,2, hypernym sense of “sheep” and “cow” are the same, namely “bovine.n.1” and “ruminant.n.1”. The word sense path of sheep.n.1 and cow.n.1 are: $\begin{matrix} Sheep . n . 1 & cow . n . 1 \\ = > bovine . n . 1 & = > bovine . n . 1 \\ = > ruminant . n . 1 & = > ruminant . n . 1 \\ = > even - toed . n . 1 & = > ungulate . n . 1 \\ = > mammal . n . 1 & = > ungulate . n . 1 \\ = > placental . n . 1 & = > placental . n . 1 \\ = > mammal . n . 1 & = > mammal . n . 1 \\ = > vertebrate . n . 1 & = > vertebrate . n . 1 \\ = > chordate . n . 1 & = > chordate . n . 1 \\ = > animal . n . 1 & = > animal . n . 1 \\ = > organism . n . 1 & = > organism . n . 1 \end{matrix}$ $\begin{matrix} = > living thing . n . 1 & = > living thing . n . 1 \\ = > object . 1 & = > object . n . 1 \\ = > physical entity . n . 1 & = > physicalentity . n . 1 \\ = > entity . n . 1 & = > entity . n . 1 \end{matrix}$

4.2 Order parameters reconstruction with word sense

Order parameter reflects similarity degree of prototype pattern vector and test pattern vector. The closer prototype pattern vector and test pattern vector, the greater corresponding order parameter, and the greater possibility of winning the competition. So construction quality of order parameter will greatly affect recognition performance of the system. Traditional synergetic pattern recognition uses pseudo-inverse method to construct order parameter, but this method requires a lot of computing power, which is without practical significance for pattern recognition of large-scale data. Some researchers consider reconstruction method of order parameter, refuse seeking of order parameter with pseudo-inverse method, but directly construct order parameter based on similarity degree of input pattern vector and prototype pattern vector, thus saving time and space.

After the introduction of word sense information, how to construct the order parameter is a key problem. Suppose the test pattern vector with word sense information is q_s, all possible roles of test pattern vector are $(r_{s}^{(1)}, r_{s}^{(2)}, \dots)$ , then order parameter ξ (q_s, r_s) of candidate role r_s can be calculated by Equation (7): $ξ (q_{s}, r_{s}) = \frac{exp \sum_{i} ω_{i} f_{i} (q_{s}, r_{s})}{\sum_{j} exp \sum_{i} ω_{i} f_{i} (q_{s}, r_{s}^{(j)})}$ (7)

Here, f_i (q_s, r_s) is feature function, ω_i represents weight of f_i (q_s, r_s). Suppose q_s = (q_s1, q_s2, …), then the expression of f_i (q_s, r_s) is: $f_{i} (q_{s}, r_{s}) = {\begin{matrix} 1, r = r_{s} and q_{s} includes attribute q_{si} \\ 0, Other \end{matrix}$

As long as ω_i can be obtained, then the semantic order parameter ξ (q_s, r_s) can be calculated by the formula (7). Estimation of ω_i may adopt Sequential conditional generalized iterative scaling algorithm (SCGIS algorithm, [32]), which seems to be very efficient. SCGIS algorithm is a model selection or parameter calculation algorithm, we can use the SCGIS algorithm to train the parameters of the model iteratively.

4.3 Network parameter reconstruction and optimization with word sense information

Dynamical evolution Equation (4) reflects dominant principle of synergetics. It can be rewritten as: $ξ_{k} = λ_{k} ξ_{k} - C ξ_{k}^{3} - (B + C) \sum_{k^{'} \neq k} ξ_{k^{'}}^{2} ξ_{k}$ (8)

In formula (8), λ_kξ_k is a self-excitation term, which represents feedback excitation of the model. $C ξ_{k}^{3}$ is self-inhibition term, which reflects the model’s inhibition of excessive growth. $(B + C) \sum_{k^{'} \neq k} ξ_{k^{'}}^{2} ξ_{k}$ is a lateral inhibition item, which reflects mutual inhibition between the models, actually a penalty term. λ_k is attention parameter, while B and C are given parameters.

Network parameter (λ_k, B, C) have important influence on the recognition performance of synergetic neural network. In this section, an efficient reconstruction and optimization algorithm of network parameter is proposed.

4.3.1 Network parameter reconstruction with the word sense information

Supposed q₁, q₂, …, q_n are the patterns remained to be recognized, a number of features f_j (q_i) , (i = 1, 2, …, n ; j = 1, 2, …, m) and g_k (q_i) , (i = 1, 2, …, n ; k = 1, 2, …, m) are extractedfrom q₁, q₂, …, q_n, where f_j (q_i) , (j = 1, 2, …, m) are the feature associated with role information and syntactic information, g_k (q_i) , (k = 1, 2, …, m) are the feature associated with word sense. n and m are the number of corresponding feature functions. Then $λ_{i} = \sum_{j} α_{j} f_{j} (q_{i}) + \sum_{k} β_{k} g_{k} (q_{i})$ (9)α_j and β_k represents the weight of feature function f_j (q_i) and g_k (q_i) respectively.

We used semantic information and word sense information to construct nework parameters, which can better describe the weight of each role chain.

4.3.2 The improved quantum particle swarm algorithm

4.3.2.1. Introduced to QPSO. Particle swarm optimization (PSO) is a population-based optimization algorithm [33, 34]. Quantum particle swarm optimization (QPSO) [28, 29] was proposed by Sun et al. in 2004, which is a probabilistic particle swarm optimization algorithm based on the quantum computation theory. It has been proved that the QPSO algorithm is a global convergence algorithm. QPSO algorithm does not need the velocity information of the particles, and has the advantages of less control parameters, simple operation and fast convergence speed. It is a general global optimization technique, which is suitable for solving all kinds of complex optimization problems. In recent years, a series of papers have focused on the application of QPSO, such as traveling salesman problems [35], sreamflow forecasting [36], RLV reentry [37] and economic load dispatch [38].

In the QPSO algorithm, the updated position of the particle is: $x_{ij} (t + 1) = p_{ij} (t) \pm \frac{1}{2} \cdot L_{ij} (t) \cdot ln (1 / - μ_{ij} (t))$ (10) $p_{ij} (t) = φ_{ij} (t) \cdot P_{ij} (t) + (1 - φ_{ij} (t)) \cdot P_{gj} (t)$ (11)

Wherein, t is the current number of iteration, x_i = (x_i1, x_i2, ⋯ , x_iD) is current position of particle i, p_i = (p_i1, p_i2, …, p_iD) is attractor of particle i in evolutionary iteration process, D indicates dimension of the problem to be solved. φ_ij (t) and μ_ij (t) are random numbers uniformly distributed in [0,1]. The control mode of L_ij (t) has a critical impact on convergence rate and performance of algorithm, It can be calculated by introducing optimum position mbest. $L_{ij} (t) = 2 \cdot β | {mbest}_{j} (t) - x_{ij} (t) |$ (12)

Wherein, β is contraction expansion factor used to adjust convergence rate of algorithm, and its calculation formula is: $β = (1 - 0.5) \times \frac{T - t}{T} + 0.5$ (13)

Here, T is the maximum number of iteration, t is the current number of iteration. Calculation formula of mbest is: ${mbest}_{j} = (\underset{D}{\underset{︸}{\frac{1}{N} \sum_{i = 1}^{N} P_{i 1}, \frac{1}{N} \sum_{i = 1}^{N} P_{i 2}, \dots, \frac{1}{N} \sum_{i = 1}^{N} P_{ij}}})$ (14)

Then final updating equation of particle:

$\begin{matrix} x_{ij} (t + 1) & = & p_{ij} (t) \pm β | {mbest}_{j} (t) - x_{ij} (t) | \\ \times ln (1 / - μ_{ij} (t)) \end{matrix}$ (15)

4.3.2.2. The improved QPSO algorithm (IQPSO). Traditional QPSO presence some defects such as easily falls into local optimum, slow convergence rate. Researchers have put forward a number of corresponding improvements. In order to maintain diversity of the population, Coelho [39] proposed a Gaussian quantum particle swarm optimization method, which realizes mutation through Gaussian probability distribution operator. Xi et al. [40] obtains optimum location mbest through weighted average, which can improve the global search ability of algorithm. These improvements in solving problems of prematurity are based on the idea of particle swarm diversity maintenance. These algorithms improve global search ability, but reduce convergence rate in varying degrees.

In quantum particle swarm algorithm, the local attractor is comprehensively decided by local optimum P_ij (t) and global optimum P_gj (t): $p_{ij} (t) = φ_{ij} (t) \cdot P_{ij} (t) + (1 - φ_{ij} (t)) \cdot P_{gj} (t)$ (16)

$φ_{ij} (t) = \frac{c_{1} r_{1}}{c_{1} r_{1} + c_{2} r_{2}}, r_{1} \sim U (0, 1), r_{2} \sim U (0, 1)$ (17)

We noted if different convergence velocity and amplitude are used in different stages, the premature convergence phenomenon can also be effectively prevented, so as to obtain better convergence performance. The two accelerating factor c₁ and c₂ can not only affect the convergence speed, but also can lead to the occurrence of premature phenomenon. When the diversity is better, c₁ should be set a little bigger, and c₂ should be set slightly smaller. When the diversity is poor, it is just theopposite.

The degree of aggregation particle swarm can be described by diversity, and the diversity of particle swarm is indicated by α = [α₁, α₂, …, α_T], wherein, α_t is adaptive diversity of t iteration particle, T is the maximum number of iteration. Population diversity concerns the average of the current position of particulate in particle swarm and optimal group position. $α_{t} = \frac{1}{N \times | L |} \sum_{i = 1}^{N} \sqrt{\sum_{d}^{D} (x_{id} - {\bar{x}}_{d})^{2}}, i = 1, 2, \dots, n$ (18)

In the formula: N represents the number of particles of particle swarm; |L| represents the maximum angular distance of search target space; x_id represents d dimension component of i particle position; D indicates dimension of the problem to be solved; ${\bar{x}}_{d}$ represents the average of d dimension component of all particle positions.

Assuming that Uc and Lc are the upper and lower bounds of the parameters respectively, then accelerating factor function can be obtained: $c_{1} (t) = {(Uc - Lc)}^{*} e^{(- 1 . 2^{*} (1 - 2 α_{t}))} + Lc$ (19) $c_{2} (t) = {(Uc - Lc)}^{*} e^{(- 0 . 4^{*} (1 - 2 α_{t}))} + Lc$ (20) The procedure of improved QPSO is given as follow:

Algorithm 1. The improved QPSO algorithm

Initialize the particle swarm;

For t = 1 to maximum iteration T

Compute the mean best position;

For i = 1 to population size M

If f(x_i) <f(P_i) then P_i = x_i; Endif

P_g = min(P_i);

For j = 1 to dimension D

update c₁ and c₂ by Equations (19) and (20);

update φ_ij (t) by Equation (17);

update p_ij (t) by Equation (16);

update x_ij (t + 1) by Equation (15);

Endfor

4.3.3 Parameters evolution

The key problem of optimizing the network parameters using intelligent optimization algorithm is the setting of the fitness function. In the SRL system, we use F1 value as the performance test index. So in our system, we can set the fitness function of the particle individual as: $f (α_{1}, α_{2}, \dots, α_{m}, β_{1}, β_{2} \dots, β_{n}, B, C) = F 1,$ wherein, (α₁, α₂, …, α_m, β₁, β₂ ⋯ , β_n, B, C) is each candidate parameter.

The reconstruction and optimization of network parameters based on improved QPSO can be described as Algorithm 2.

Algorithm 2. Reconstruction and optimization of network parameters based on improved QPSO

1) Role-related features and word sense-related features are used to construct semantic vector space model;

2) Obtain feature vectors from the train corpus and test corpus, construct test pattern q_l (l = 1, 2, …) and possible roles r_k (k = 1, 2, …);

3) Combination of all possible roles of q_l (l = 1, 2, …), obtain all the possible roles chains R_i (r_i1, r_i2, …);

4) Set λ_i by Equation (9);

5) Obtain the optimal solution of parameters (α₁, α₂, …, α_m, β₁, β₂ ⋯ , β_n, B, C) based on improved QPSO algorithm;

6) Marked role chain is ultimately obtained through iterative competition of semantic order parameters in dynamics equation.

5 Experiment

5.1 Data description

In the experiments, the OntoNotes Release 2.0 corpus [41] is used to test the performance of semantic role labeling. The corpus has been annotated with predicate argument structure and word senses in three languages, and provide a chance for us to get word senses information and semantic role information. OntoNotes have seven data sets: ABC,CNN,MNB, NBC,PRI,VOA and WSJ. For each data set, we approximately take 60% of the training set / 20% of the development set/20% of test set. To obtain word sense information, corpus must be first preprocessed.

(1) Syntax Conversion

OntoNotes 2.0 corpus adopts phrase structure-based syntactic analysis. To study semantic role labeling based on dependency, the phrase-based syntactic structure must be translated into dependency-based syntactic structure. We adopt Constituent-to-Dependency Coversion Tool Kit for conversion.

(2) Word sense Information

In OnteNote 2.0 corpus, many words are provided in plural or past tense form. There is no corresponding lexical information, nor word sense information. This information can be obtained by calling API function provided by WordNet 2.1.

5.2 Experiment results

In order to carry out a detailed comparison, we use six strategies.

SVM: semantic role labeling based on support vector machine.

SNN: semantic role labeling based on SNN.

SNN+BWS: SNN-based semantic role labeling with basic word sense information (Lemma+Sense).

SNN+BWS+MWS: SNN-based semanticrole labeling with word sense information (Lemma+Sense, Hypersense(1), Hypersense(2), Hypersense(3), Lemma+Hypersense(1), Lemma+Hypersense(2), and Lemma+Hypersense(3)).

SNN+IQPSO: SNN-based semantic role labeling based on improved QPSO.

SNN+BWS+NWS+IQPSO: semantic role labeling with word sense information base on improved SNN.

The comparison of different methods are shown in Tables 1 to 6.

As can be seen from Table 1, F1 value obtained of 7 kinds of data sets from SNN model are better than the results based on SVM-based SRL model. The experiment with biggest upgrade is MNB corpus, with F1 value increased by 0.45 (84.76-84.31), and the remaining corpus with smallest upgrade of F1 value is NBC corpus, with F1 value increased by 0.09 (81.57-81.48). Seen from recall rate, recall rate obtained from VOA corpus and WSJ corpus is inferior to SVM-based SRL model, but the difference is small. Experimental results show that the method to obtain optimal roles chain through order parameter evolution is effective, which can improve the performance of semantic role labeling system.

As can be seen from Table 2, for all seven corpus, after adding basic word sense information, the system performance is improved. The experiment with biggest upgrade is ABC corpus, with F1 value increased by 0.53 (85.25-84.72), and the remaining corpus with smallest upgrade of F1 value is WSJ corpus, with F1 value increased by 0.05 (84.54-84.49).

To further discuss the role of word sense information, we add Lemma+Sense, Hypersense(1), Hypersense(2), Hypersense(3), Lexical+Hypersense(1), Lexical+Hypersense(2), and Lexical+Hypersense(3) in the experiment. As can be seen from experiment results of Table 3, after addition of these semantic features, the performance is further improved. It can be seen that word sense information can better describe semantic relations that cannot be reflected by lexical information and part of speech information, thus improving the performance of semantic role labeling system.

To further analyze impact of word sense-related features on semantic role labeling performance, we made further experiments on all corpus (Test set of all seven kinds of data sets as experimental test set). Table 4 shows performance comparison of different word sense features. The experiment with biggest upgrade is Hypersense(1), and the remaining corpus with smallest upgrade of F1 value is Hypersense(3).

From Table 5, we can see that the performance of SNN+IQPSO outperforms SNN model, and F1 value obtained in seven kinds of data sets is better than SNN model. The experiment with biggest upgrade is ABC, with F1 value increased by 0.62 (85.34-84.72), and the remaining corpus with smallest upgrade of F1 value is MNB, with F1 value increased by 0.12 (81.79-81.57). This demonstrates that optimization of model parameters is very important and to obtain model parameters through optimization algorithm can further improve system performance.

As can be seen from Table 6, compared to SVM-based SRL model, SNN+BWS+NWS+IQPSO enjoys better semantic role labeling performance. F1 values obtained from seven kinds of corpus are improved to some extent. The proposed models in this paper significantly enhance the efficiency and effectiveness of semantic analysis.

Figure 3 shows convergence curve of three different data sets, which indicates that IQPSO algorithm can more effectively select model parameters of synergetic semantic analysis model, so as improve performance of SRL system.

6 Conclusion

An improved SNN-based SRL model with word sense information is proposed and implemented in this paper. Experiments show the synergetic semantic analysis have a higher performance for SRL.

We got the following conclusions.

The improved SNN model can effectively utilize word sense information, which can further improve the performance of semantic analysis.

The improved QPSO algorithm based on adaptive accelerating factor selection has a better global convergence ability, and can effectively regulate the parameters of semantic analysis model.

In the next work, we will consider the use of deep learning [42] to construct the order parameter of SNN, and thus better describe the semantic relations in the semantic chain.

Footnotes

Acknowledgments

This work was supported by the Science and technology project of Fujian Provincial Education Department (Grant No. JA15026) and the Quanzhou science and technology project (Grant No. 2013Z17 and 2015Z113).

References

Punyakanok

, Roth

and Yih

W.T.

, The importance of syntactic parsing and inference in semantic role labeling, Computational Linguistics34(2) (2008), 257–287.

Toutanova

, Haghighi

and Manning

C.D.

, A global joint model for semantic role labeling, Computational Linguistics34(2) (1974), 161–191.

and Pascale

, Can semantic role labeling improve SMT?In Proceedings of EAMT, 2009, pp. 218–225.

Shen

and Lapata

, Using semantic roles to improve question answering, In Proceedings of EMNLP, 2007, pp. 12–21.

Narayanan

and Harabagiu

, Question answering based on semantic structures, Proceedings of the International Conference on Computational Linguistics (COLING), 2004, pp. 693–701.

Surdeanu

, Harabagiu

, Williams

and Aarseth

, Using predicate-argument structures for information extraction, ACL ’03 Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, 1, 2003, 8–15.

Surdeanu

, Johansson

, Meyers

, Marquez

and Nivre

, The CoNLL-shared task on joint parsing of syntactic and semantic dependencies, Twelfth Conference on Computational Natural Language Learning, 2008, pp. 159–177.

Meza-Ruiz

and Riedel

, Jointly identifying predicates, arguments and senses using markov logic, Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, 2009, pp. 155–163.

Che

W.X.

, Liu

and Li

Y.Q.

, Improving semantic role labeling with word sense, Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, 2010, pp. 246–249.

10.

Lim

, Lee

and Ra

, Dependency-based semantic role labeling using sequence labeling with a structural SVM, Pattern Recognition Letters34(6) (2013), 696–702.

11.

Zhao

, Zhang

and Kit

, Integrative semantic dependency parsing via efficient large-scale feature selection, Journal of Artificial Intelligence Research46(1) (2014), 203–233.

12.

J.H.

, Zhou

G.D.

, Zhu

Q.M.

and Qian

P.D.

, Semantic role labeling in Chinese language for nominal predicates, Journal of Software22(8) (2011), 1725–1737.

13.

Nielsen

R.D.

and Pradhan

, Mixing weak learners in semantic parsin, In Proceedings of EMNLP-2004, 2004, pp. 80–87.

14.

Màrquez

, Comas

, Giménez

and Catala

, Semantic role labeling as sequential tagging, Proceedings of CoNLL-2005 Ann Arbor, Michigan, 2005, pp. 193–196.

15.

Haken

, Synergetic Computers and cognition–A top-down approach to neural Nets, Springer-Verlag, Berlin, 1991.

16.

Shao

, Gao

and Yang

X.Z.

, Synergetic face recognition algorithm based on ICA, In Proceedings of the International Conference on Neural Networks and Brain, Beijing, China, 2005, pp. 249–253.

17.

Tong

, Tang

L.H.

, Ip

H.H.S.

and Qi

, On relevance feedback and similarity measure for image retrieval with synergetic neural nets, Neurocomputing51(2) (2003), 105–124.

18.

Huang

Z.H.

, Improved quantum particle swarm optimization for mangroves classification, Journal of Sensors2016 (2016), 8. Article ID 9264690.

19.

Dang

H.T.

and Palmer

, The role of semantic roles in disambiguating verb senses, Proceedings of Annual Meeting of the Association for Computational Linguistics Ann Arbor Mi, 2005, pp. 26–28.

20.

Jan

Hajič

, Ciaramita

M.,

, Johansson

, Kawahara

, Martí

M.A

, Màrquez

, Meyers

, Nivre

, Padó

, Stěpánek

, Straňák

, Surdeanu

, Xue

and Zhang

, The conll-shared task: Syntactic and semantic dependencies in multiple languages, Proceedings of CoNLL, 2009, pp. 1–18.

21.

Dahlmeier

, Ng

H.T.

and Schultz

, Joint learning of preposition senses and semantic roles of prepositional phrases, Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, EMNLP, 2009, pp. 450–458.

22.

Che

W.X.

and Liu

, Jointly modeling wsd and srl with markov logic, Proceedings of the 23rd International Conference on Computational Linguistics. Association for Computational Linguistics, 2010, pp. 161–169.

23.

Fang

, Liu

and Wang

, An improved attention parameter setting algorithm based on award learning mechanism, Computerence29(11) (2002), 195–197.

24.

X.L.

and Jiao

L.C.

, Reconstruction of order parameters based on immunity clonal strategy for image classification, In Camilho

Aurélio

, Kamel

Mohamed

, (Eds.), Lecture Notes in Computer Science, Springer, Berlin, 3211 (2004), pp. 455–462.

25.

Zou

, Ao

Y.H.

, Yao

and Sun

J.X.

, An unbalanced parameters optimization of synergetic neural network based on differential evolution algorithm, Chinese Journal of Signal Processing26(12) (2010), 1768–1773.

26.

Jiang

Z.H.

and Dougal

R.A.

, Synergetic control of pwer converters for pulse current charging of advanced batteries from a fuel cell power source, IEEE Transactions on Power Electronics19(4) (2004), 1140–1150.

27.

Gao

, Dong

H.M.

, Shao

and Zhao

, Parameters optimization of synergetic recognition approach, Chinese Journal of Electronics14(2) (2005), 192–197.

28.

Sun

, Feng

and Xu

, Particle swarm optimization with particles hng quantum behavior, Proceedings of the Congress on Evolutionary Computation70 (2004), 325–331.

29.

Zhan

, Lu

, Hao

and Jin

, Improving particle swarm optimization: Using neighbor heuristic and gaussian cloud learning, Intelligent Data Analysis20(1) (2016), 167–182.

30.

Wang

H.L.

and Zhou

G.D.

, Semantic role labeling of Chinese nominal predicates with dependency-driven constituent parse tree structure, Journal of Computer Science and Technology28(6) (2013), 1117–1126.

31.

Pradhan

, Hacioglu

, Krugler

, Ward

, Martin

and Jurafsky

, Support vector learning for semantic argument classification, Machine Learning60(1-3) (2005), 11–39.

32.

Goodman

, Sequential conditional generalized iterative scaling, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 2002, pp. 9–16.

33.

Zhan

, Lu

, Hao

and Jin

, Improving particle swarm optimization: Using neighbor heuristic and gaussian cloud learning, Intelligent Data Analysis20(1) (2016), 167–182.

34.

Bagheri

, Miri

and Shabakhty

, Fuzzy reliability analysis using a new alpha level set optimization ap-proach based on particle swarm optimization, Journal of Intelligent & Fuzzy Systems30(1) (2015), 235–244.

35.

Yao

W.Q.

, Genetic Quantum particle swarm optimization algorithm for solving traveling salesman problems, Advances in Intelligent Systems & Computing (2014), 67–74.

36.

, Anand

, Panigrah

B.K.

and Mathur

, Streamflow forecasting by svm with quantum behaved particle swarm optimization, Neurocomputing101(3) (2013), 18–23.

37.

Zhang

, Wang

, Tao

, Sarker

and Cheng

, Design of optimal attack-Angle for RLV reentry based on quantum particle swarm optimization, Advances in Mechanical Engineering2 (2014), 677–681.

38.

Hosseinnezhad

, Rafiee

, Ahmadian

and Ameli

M.T.

, Species-based quantum particle swarm optimization for economic load dispatch, International Journal of Electrical Power & Energy Systems63(12) (2014), 311–322.

39.

Coelho

L.S.

, Gaussian quantum-behaved particle swarm optimization approaches for constrained engineering design problem, Expert Systems with Applications37(2) (2010), 1676–1683.

40.

, Sun

and Xu

, An improved quantum-behaved particle swarm optimization algorithm with weighted mean best position, Applied Mathematics and Computation205(2) (2008), 751–759.

41.

Hovy

, Marcus

, Palmer

, Ramshaw

and Weischedel

, Ontonotes: The 90% solution, In Proceedings of the human language technology conference of the NAACL, 2006, pp. 57–60.

42.

Bengio

, Learning deep architectures for AI, Foundations & Trends in Machine Learning2(1) (2009), 1–127.