A graph model based feature set selection from short texts with application to document novelty detection

Abstract

Document novelty detection is a concept learning problem wherein the system gains its knowledge only from the positive documents under a concept and with that limited knowledge it attempts to detect the negative cases. This work focuses on learning author style as a concept from the given set of documents, particularly emails. Since author attribution for shorter texts such as emails is more complex compared to larger documents, the techniques originally used for the large documents prove inefficient for short texts. To address this shortcoming of existing algorithms in detecting aberration in author style, we have proposed a graph-model based technique for feature set extraction from short documents. Given the extracted feature set, we have also developed two probability based text representation schemes that could best represent a text document to an underlying one-class SVM classifier. The proposed models have been compared and evaluated on the public Enron email dataset. Applying graph based feature set extraction technique in combination with the inclusive compound probability based text representation has proved to be very efficient. The generality of the proposed method allows the approach to be applicable to all kind of text documents including emails.

Keywords

Novelty detection author attribution graph model feature extraction one class SVM email classification

1. Introduction

Equipped only with the knowledge of known data, a system can achieve the ability to differentiate between known and unknown instances through novelty detection mechanisms. This becomes extremely important in the real life scenarios such as fault detection or intrusion detection systems, where it is nearly impossible to know in advance about all possible kinds of data the system would encounter. Novelty detection has proven to be a very challenging task due to the difficulty in defining a decision boundary for the known class, pertaining to absence of negative instances. Classifiers that are designed to detect whether an input is a known data or unknown are called one-class classifiers. Our goal is to build a one-class classifier that could perform novelty detection in short documents for authorship attribution.

The idea behind novelty detection in documents is to learn an inherent concept from a set of related documents. The concept could be any knowledge that relates all the documents under the known class such as document category, topic of interest, sentiment of the article, author style, etc. In this paper, we deal with learning the writing style of an author from a given set of documents written by him/her. The field of research related to author style classification is generally termed as author attribution [1]. So the focus of our research work is the intersection of novelty detection and author attribution, wherein we try to detect if an anonymous document is written by the particular author or not, by gaining the required knowledge only from the author’s archived documents.

A lot of research has been performed for document author attribution in the past, focusing on larger documents such as books or lengthy articles. We have observed that these mechanisms when applied on documents of smaller size do not perform efficiently. Also most of the works have approached author attribution as a multi-class classification problem, wherein the system gains the required knowledge on the style of a given set of authors from a sample set of their documents and tries to classify an anonymous document to any of the authors. However in case of novelty detection, we need a single-class classification mechanism which could gain knowledge of a particular author’s style only from the set of sample documents written by that author, without any counter example.

We have based this research work on emails, as a representation of small sized documents. However, the proposed classification model can be applied to any text content. We have attempted to build a knowledgeable system that could learn the author’s style from the old emails of a person and could identify if a given new email text follows the author’s writing style or not. Several challenges had to be addressed in identifying authorship for an email. First of all, emails are generally concise in nature, with some tens to hundreds of words. Moreover, unlike books or other large texts, the vocabulary of an email acutely changes based on the context or the addressee. To overcome these issues, our work focused on identifying the ‘writer invariant’ features, which are used by the author subconsciously, so that the usage of these features is independent of the context or addressee and it cannot be manipulated. The significant contributions of this paper are as follows:

•
A graph-model based algorithm has been proposed for feature set selection. A comprehensive graph is constructed to represent the old emails and the most significant features are selected from this graph. This process results mostly in the writer-invariant features.
•
Two text representation schemes have been proposed to generate feature vectors for documents, and are subsequently used by the machine learning algorithm, such as One-Class Support Vector Machine. The first is Probability Model (from here on, referred as PM), wherein the value of each feature is the probability with which the feature appears in that email. The second one is Inclusive Compound Probability Model (from here on, referred as ICPM), where the value of each feature is the probability with which the feature appears in a sentence of the email.
•
The proposed models have been implemented and tested against the public Enron dataset. Since Enron corpus is the only substantial collection of real emails that is public, by testing our prototypes with this corpus, we could measure the performance of our approach in a real-life scenario.
•
We have compared the proposed models with other author attribution and novelty detection techniques. ICPM has proved to be the most efficient. We have evaluated the performance of ICPM against all possible values for its parameters and attempted to arrive at optimal values.

The rest of this paper has been organized as follows. Section 2 describes the previous works related to novelty detection, stylometry, author attribution and email categorization. Section 3 explains the proposed model in detail. Section 4 shows the performance of the proposed model compared to existing techniques and then presents an evaluation of the proposed model. Finally, in Sections 5, we suggest possible applications for the proposed classification model and conclude by stating the way forward.
2. Background

Novelty detection in documents for author attribution is a research problem connected with many existing research fields such as author attribution, stylometry, email classification and novelty detection. Work on the field of authorship attribution has been carried on for hundreds of years. During 19th century, several non-traditional methodologies have been proposed to prove the authorship of a given text. These techniques used to be statistical in nature and were referred to as stylometry [2].

In 1851, Augustus de Morgan has suggested that the length of words could be used as a differentiator for various author styles in Bible [2]. Medelhall published his investigation results in 1887 to prove that word lengths are not significant after working on lengths of several hundred thousand words from the works of Bacon, Marlowe, and Shakespeare. However, he did find that the works of Shakespeare and Marlowe had similarities but they were consistently different from Bacon [1].

In 1932, George Zipf worked on stylometry and proposed a universal law of language, “Statistical rank of a word varies inversely to its frequency” [3, 4]. Yule in 1944, define “Yules characteristic K”, as a measure of word frequencies, the basic assumption being that the occurrence of a word can be modeled by a Poisson distribution [3]. This marker was also proved wrong by subsequent researches in this area. In 1963, Mosteller and Wallace resolved the conflict with the authorship of Federalist Papers, using statistical methods [3, 4].

It was then that the significance of function words came to focus. Function words are words such as ‘to’, ‘if’, ‘by’, ‘upon’ and so on, that have little lexical meaning, but instead serve to express grammatical relationships with other words in the sentence. The pattern of usage of such words shows the attitude of the author or the author’s particular language style. Subsequently, a lot of research has been carried out on this area. The basic strategy is to train a model with several documents of the authors in consideration, then the given sample text is checked against the model to identify its author [1, 2].

Feature selection has played an important part in the performance of the algorithms. The selected features were statistics on usage of sentences, words, and special characters. Roughly they can be classified as i) character features such as character $n$ -grams or special characters [5, 6], ii) lexical features such as word $n$ -grams or functional words [4, 5, 6, 7] and iii) syntactic features such as $n$ -grams of POS tags and rewrite rules [5]. Some works have tried to optimize the feature selection process using techniques such as Principal Component Analysis [4, 7].

Text representation is another aspect which plays a major role in classification performance. This deals with how the classifier understands the samples. The features selected by the Feature set selection procedure are converted to real numbers by some kind of transformation so that the machine learning algorithm could understand them. Many representations such as binary, frequency, TF-IDF, hadamard representation have been used in several previous works on document classification [8, 9, 10, 11].

Classifier is the core component of all classification algorithms. A classifier is trained in the beginning of the algorithms with sample documents/text of the concerned authors. After the training phase, the classifier gains enough knowledge on the authors’ styles, so that it can identify the author of a given text. Different classifiers have been used such as Naïve Bayes [6], Support Vector Machine [11, 12], Neural networks [7] for author attribution in the past.

Books, legal documents or large text have been the subject of research in most of these works. Very less work has been done to identify authorship of small text such as emails, since there are several challenges in email author attribution as outlined in the previous section. Categorization of emails to custom folders was the focus of several works, such as using set theory [13], graph-based approach [14] or language models [15] for classification. Spam mails identification is another class of work done on emails [13]. De Vel et al. used attributes such as style markers as well as structured features [16]. It also used statistical count of vocabulary, functional words and number of short words for forensic investigations. Similar features were used to identify gender of the author, with the assumption that women’s language makes more frequent use of emotionally intensive adverbs and adjectives and that their language is more punctuated. Whereas, men’s language was assumed to have characteristics such as strong assertions, aggressive, self-promotion, rhetorical questions, authoritative orientation, challenges and humor [17]. Goodman et al. used a set of stylistic features to determine authorship. Statistics collected on these features such as number of words, number of sentences, number of special characters, etc. were used for further KNN or SVM classification [18, 19].

Most of the above mentioned works deal with multi-class classification. But novelty detection is all about one-class classification. Novelty detection focuses on how to find out any abnormality of a negative sample under a predefined condition. Here again, classifier forms the core component. Variants of several multi-class classifiers, such as Naïve Bayes [20], Support Vector Machine [11, 21], Neural networks [10, 21, 22] have been developed to deal with single class classification.

To the best of our knowledge, there has not been any novelty detection work over short text documents such as emails. Also, author attribution has not been or tested over a public email dataset. The main motivation of this work is to address these two areas.

3. The graph model based classifier

3.1 Architecture overview

As mentioned in the previous sections, novelty detection is a subset of classification problem, wherein the number of target classes is just one. As with any text classification algorithm, a 3-step procedure is followed: i) Feature set selection, ii) Training phase and iii) Testing phase. The proposed classification model analyses the lexical features of the email texts to learn the hidden pattern. This needs the raw email text to be pre-processed and converted into lexical tokens before being fed to any of these steps.

Pre-processing of text is a fundamental operation in text classification. This operation cleans unwanted characters or contents from the text and tokenizes the useful content of the text for further processing. As lexical features are the only features of interest, all special characters are stripped off in the pre-processing stage. Also, the text is converted to lower case, thereby removing the distinction between upper-cased and lower-cased words. All the email texts, both in training phase and in testing phase are preprocessed, tokenized and then forwarded to rest of the processing.

Feature-set selection is a critical phase in classification models. Efficiency of a model greatly depends on the features selected to identify the pattern. Here, all emails under the training set are converted to a comprehensive graph and the significant feature-set is identified. The detailed process will be explained in Section 3.2.

Classifier is the core component used in both the training and testing phases. In the context of prevention/detection of email impersonation, we need a classifier that could only be trained with positive samples. There are various approaches proposed to this problem in previous works, such as neural networks, one-class SVM and so on. We have opted to use one-class SVM classifier due to its performance advantage. Based on the technique used for text representation, we have defined two classification models – Probability Model (PM) and Inclusive Compound Probability Model (ICPM). Under training phase the one-class SVM classifier is trained with the feature vectors corresponding to the feature-set of each email in the training set. The classifier is fed with the feature vector of a new unclassified email in the testing phase to detect its legitimacy. Section 3.3 will provide details on the text representation schemes and Section 3.4 explains the underlying classifier algorithms.

Depending on the application requirement, the model parameters could be tuned to configure either a specific model or a sensitive model. A specific model could achieve high Specificity/True Negative Rate (TNR) and a sensitive model could provide high Sensitivity/True Positive Rate (TPR). TPR defines how effectively the model identifies the positive (author’s own) emails. TNR represents the model’s performance in identifying the illegitimate emails. Currently, a model with both high specificity and high sensitivity is not possible, as there is a trade-off between the two. More on this trade-off will be covered in the Results section.

3.2 Feature set selection

Feature-set selection is the process of selecting the set of features that best define the model, and in this case, these are the features that best define the author’s writing style. In the lexical view, words and $n$ -gram words form the candidate features. Finding the optimal set of features from the list of candidate features is the objective of this step. There is a trade-off in determining the size of the feature set. Having a very small feature set may make it impossible to formulate a good hypothesis. However, a very large feature set may add more noise and overfit the data [8]. Appropriate feature size needs to be experimentally determined based on the application requirement.

The process begins by converting the pre-processed documents into tokens of words. This conversion is done for all the documents under the given training set and merged to a comprehensive representation.

All of these tokens form feature set candidates and passed through an algorithm which could identify the set of features that could uniquely identify the author. Other than the word tokens, there are other derived parameters which could be considered as features. Stylometry techniques consider punctuation marks and statistics such as number of sentences, number of paragraphs, etc as features [18]. Combination of n adjacent words could be considered as features [23]. In this work we have come up with a technique to generate features from a graph model, where the nodes and links of the graph form the feature set candidates. The following paragraphs explain various existing techniques that could be employed to select the feature set and then the proposed method.

Frequency method.

One of the simplest ways for feature set selection is the frequency method. All the words are considered to be candidate features in this method. Feature set extraction involves filtering out the least frequently used words from the list of candidate features. This results in a set of words which are used more often by the author. More often a word is used by the author, greater is its probability to be part of the feature set. The frequency threshold used to divide the most frequently and least frequently used features is usually evaluated experimentally based on the performance of the underlying classifier.

TF-IDF method.

Information retrieval algorithms treat the most frequent words as stop words as they do not carry any significance. They state to give more importance to the rare words as they have more relation to the topic/category under consideration. So TF-IDF scheme was introduced, where TF is the term frequency and IDF is inverse document frequency [8]. IDF of a term $t$ is calculated using the formula in Eq. (1). Number of documents in which the given term $t$ appears is denoted by $df_{t}$ and $N$ is the number of documents in the sample space. The value of $idf_{t}$ is less for frequent words and more for rare words, as the document frequency is used in the denominator.

$idf_{t}=log\left(\frac{N}{df_{t}}+1\right)$ (1)

TF-IDF is the combination of both these values to arrive at the overall weight of each term and is calculated by multiplying TF of a term to its corresponding IDF. In TFIDF method [8], terms with less TF-IDF value are discarded and the threshold used to eliminate less weight terms is determined experimentally.

Graph-model based approach.

In this work, we propose a graph-model based feature set selection technique, wherein the tokens are adapted to a graphical representation, with words represented as nodes and adjacent nodes are connected with links. Rest of this subsection is dedicated to explain the existing graph representation methods [24] and the proposed n-simple distance $+$ frequency representation.

Under a general graph representation scheme, vertices and edges are used to represent a text document in the form of a graph. Each vertex in the graph represents a unique word in the document. If the word a is followed by a word b in the given section s of a document, it is represented as a directed edge from the node a to b in the graph. A graph $G$ is defined as ( $V$ , $E$ , $\alpha$ , $\beta$ ), where $V$ is the set of vertices, $E$ is the set of edges between the vertices, $\alpha$ is the weight function corresponding to vertices and $\beta$ is the weight function for edges. The different approaches for graph representation vary mainly on $E$ , $\alpha$ and $\beta$ . Based on the factors such as how the frequency of occurrence of a word is represented in the graph ( $\alpha$ ) and the criteria on which two nodes are connected with a directed edge ( $E$ ), various graph representations have been suggested in [24] as given below.

•

Standard and Simple representations: These are the simple forms, where each term is represented as a vertex and edge is created between two adjacent words. Irrespective of the number of times a word appears in the document, only one node is created to represent the word.

•

$n$ -distance representation: A user defined parameter n is used. For each term, every succeeding term upto a distance n are looked ahead and edges are created between them. The edges are labeled with its corresponding number of occurrence.

•

$n$ -simple distance representation: This representation also uses a user defined parameter n. For each term, every succeeding term upto a distance n are looked ahead and edges are created between them. The edge does not have a label on its distance from the source node, rather it only means that the distance between the two connected terms is not more than n. The advantage of this scheme is that it does not only identify the words that are placed next to each other, but also helps in identifying the relationship between the words that are used in combination within a distance of n. When n becomes the sentence length, the graph model could depict the relation between words that are always used together in sentences. In our case, if n is set to $-$ 1, the distance is considered to be the sentence length and edges are created between all the words in the sentence in the forward order and we refer to such a structure as fully-connected structure.

•

Frequency representation: In this representation, each node and edge is labeled with their corresponding frequency. For nodes it indicates how many times the associated word has appeared in the document. For edges it indicates how many times the connected terms have appeared adjacent to each other in the specified order. The benefit of this scheme is that it helps in identifying the significance of a node or edge on the author’s style.

In our work, we propose a new graph representation scheme by combining both n-simple distance and frequency representations in order to tap the advantages of both the schemes and hence effectively represent emails in the form of graphs. We call this scheme as n-simple distance $+$ frequency representation. In this representation, all the terms within a distance of n are connected by an edge and each node/edge have a frequency label associated with it, to keep track of the number of times it appears in the document.

Figure 1.

Conversion of email text to a graph.

Figure 1 shows how this scheme is used to convert an email text to a fully-connected graph structure. Frequency of the nodes is shown within the node’s circle and that of the edges are shown next to the edge. The node “thanks” has a frequency 2, as it has appeared twice in the email. And all the remaining nodes have a frequency 1 as they appear just once. Since this is a fully connected structure, each node is connected to all the nodes representing successive words in the sentence.

In a similar fashion, all emails under the training set are converted into graphs. These graphs are merged to form a consolidated graph, wherein the frequency of a node/link represents the frequency of the same in all the mails together. The consolidated graph provides a rough view of the overall words usage. For example, the number of nodes and links used to represent a particular user (refer Section 4.1 for details) with around 600 emails was roughly 2378 nodes and 50892 links. If the nodes and links are sorted based on their frequency, then we could safely say that the topmost nodes/links significantly dictate the author’s style. Hence, the top n% of nodes and top m% of links have been selected as the feature set, to be used for the classifier.

Figure 2.

Top 25 nodes from a user model.

The challenge with this strategy is arriving at a precise value for the parameters m and n. It has been found experimentally that the ideal value of n for nodes is in the range 1–5% and the ideal value of m is of the range 0.5–2.5% (refer Section 4.4). Even with lesser feature set, classification is possible, with a drop in performance. More on the effect of feature size on performance is explained in detail in the Experimental Results section. To appreciate the significance of this strategy, consider the top 25 nodes from a particular user’s model depicted in Fig. 2. As it can be seen, the top 25 nodes of the model mostly consists of function words (‘the’, ‘is’, ‘or’, ‘of’, ‘on’, ‘and’, ‘to’, etc.) and certain generally used key words (‘lynn’) and these words are mostly not related to any specific topic or context. They correspond to the author’s style about which even the author may not be aware and they do not change with underlying topic or context.

3.3 Text representation

Text documents need to be converted to an appropriate format which could be recognized by the classifier component. For any classifier using machine learning, probabilistic or heuristic algorithms, the recognizable format is that of a vector of real numbers. The text of a document is transformed to a feature vector of real numbers corresponding to the feature set, using some text representation techniques. Below are listed some of the schemes used for text representation and in the later part we explain the proposed schemes in detail.

Binary representation.

This is the simplest technique used for text representation. Here, the presence of a feature in a document is denoted with a 1 and its absence by 0. The feature vector is thus a vector of 0’s and 1’s. The frequency of occurrence of a feature is not captured in this representation. Thus a value of 1 corresponding to a feature in the feature vector means that the feature is in the document.

Frequency representation.

Converse to the previous scheme, this scheme gives importance to the number of times a feature occurs in a document. Each feature is given a value that is equal to the frequency of occurrence of the feature in the document. The feature vector is thus composed of the frequency of each feature in the document. Here, a value $f$ corresponding to a feature in the feature vector means that the feature has occurred $f$ times in the document.

TF-IDF representation.

Since most used words get more weight in frequency representation, function words end up getting more significance in that scheme. However, in most of the information retrieval algorithms, such words are treated as stop words and are simply discarded. So, in order to give more weight to rare words than such stop words and also to make use of the document frequency in determining weight, TF-IDF representation was introduced for text representation. IDF when combined with TF compensates the frequency of stop words and boosts the weight of rare words.

Hadamard representation.

This scheme was used extensively in [10, 11], where feature vectors are populated with the Hadamard product of the frequency of a feature in the document to that of the feature in the whole training set. If e is the frequency vector of all features in a given document and E is the frequency vector for the complete training set, then Hadamard product is the component-wise product of these two vectors computed using the formula in Eq. (2).

$\vec{d}=\vec{e}\otimes\vec{E}$ (2)

Probability representations.

In this paper, we propose to make use of the probability of occurrence of the features to represent them in feature vector. This resulted in two types of models, which will be explained in the following paragraphs. Both the schemes are explained here in context with the graph model.

3.3.1 Probability model (PM)

In this model, the value of a feature in a given email is the probability of occurrence of the feature in the email. This model treats the whole email as a bag of features. For a feature node, it is the ratio of frequency of the corresponding word to total number of words in the email. The same computation applies for feature links as well. Mathematically, the probability of a feature node (Fnode) and feature link (Flink) are computed using Eqs (3) and (4), where n is the total number of nodes and l is the total number of links in the given email.

$\displaystyle\textit{P(Fnode)}=\frac{\textit{frequency(Fnode)}}{\sum\limits_{i% =0}^{n}\ \textit{frequency}(\textit{node}_{i})}$ (3) $\displaystyle\textit{P(Flink)}=\frac{\textit{frequency(Flink)}}{\sum\limits_{i% =0}^{l}\ \textit{frequency}(\textit{link}_{i})}$ (4)

3.3.2 Inclusive compound probability model (ICPM)

This model assigns value to a feature based on the probability of occurrence of the feature in a sentence in the email. Here, the attempt is to capture the pattern of feature usage in a sentence. The occurrence of a feature in two difference sentences of an email is mutually inclusive. Hence this is a case of inclusive compound probability (ICP). If $P(f_{\text{A}}$ ) is the probability of a feature to be in sentence A and $P(f_{\text{B}}$ ) is the probability of the feature to be in sentence B, then the probability of the feature to occur in any of the sentences in an email can be computed by Eq. (5).

$P(f)=P(f_{A})+P(f_{B})-P(f_{A})P(f_{B})$ (5)

3.4 Classifier algorithms

Classifier learns the pattern of the input feature vectors during training phase. Then using the gained knowledge, the classifier finds out the class to which a test sample belongs. Most of the classifiers were originally designed for multi-class classifications. With certain extended logic, these algorithms have been made to work with one-class classifications.

Prototype algorithm.

This is the Rocchio’s method of classification [8]. The basic idea in this algorithm is to represent each document as a vector $\vec{d}$ in a vector space, so that documents with similar content will have similar vectors. This method follows the TF-IDF representation explained in the previous section to transform text documents to document vectors. So, each element is a combination of the statistics $tf(w)$ and $idf(w)$ of the corresponding word $w$ . The class model for the training set is obtained by combining the document vectors into a prototype vector $\vec{c}$ for the class $C$ using Eq. (6).

$\vec{c}=\sum\limits_{d\in{C}}\vec{d}$ (6)

The maximum distance $\delta$ of all the samples in training set to the prototype vector $\vec{c}$ is determined. Any test sample that has a distance less that $\delta$ to $\vec{c}$ are considered to be part of the class $C$ .

Nearest neighbor algorithm.

This is another algorithm for single-class classification achieved by altering the basic nearest neighbor algorithm for multi-class. The maximum distance $\delta$ that any sample in the training set has to any other sample in the same set is determined. If the test sample lies within the distance $\delta$ to all the samples in training set, then the test sample is considered to be part of the class.

Naïve Bayes method.

This algorithm uses the probabilistic details of features in the document to determine its class. The probability of a document d to belong to the class $C$ , $p(d|C)$ is calculated by the product of the probability of all of its features to belong to the class $C$ , $p(f|C)$ . The threshold probability is calculated to be the minimum probability that a document in training set to the class $C$ . Any test sample having a probability greater than the threshold probability is marked as part of the class.

Autoencoder.

This is a one-class classifier based on neural networks [21, 22]. The network is designed with three layers. The input and output layers have same number of nodes as the number of features in the feature set. The hidden layer has less number of nodes compared to the other layer, thereby forming a bottleneck forcing the network to learn the identity function to reconstruct output from input. If the network is able to reconstruct the test sample successfully from the input test sample vector, then the sample is classified to belong to the trained class.

One class SVM.

In this work, we have used one-class SVM following Scholkopf methodology [25]. This again is an enhancement on top of the two-class SVM. In this methodology, after transforming the features via kernel, they treat the origin as the only member of the second class. Then using some relaxation parameters, they separate the image of one class from the origin. Gradually the SVM learns the boundary of the given class.

The main idea of this algorithm is to return a function $f$ that takes the value $+$ 1 in a small region capturing most of the data points, and $-$ 1 elsewhere. The strategy is to map the data into the feature space corresponding to the kernel, and to separate them from the origin with maximum margin. For a new point $x$ , the value $f(x)$ is determined by evaluating which side of the hyperplane it falls on, in feature space. To separate the data set from the origin, we solve the following quadratic equation,

$\min{\frac{1}{2}\lVert{w}\rVert^{2}+\frac{1}{vl}\sum\limits_{i=1}^{l}\xi_{i}-\rho}$

subject to $(\omega.\phi(x_{i}))\geqslant\rho-\xi_{i},i=1,2,\ldots l\ \xi_{i}\geqslant 0$ .

where, $l$ is the size of the training set; $0\leqslant\nu\leqslant 1$ ; $\nu$ is the parameter that determines how tight the class boundary should fit the data.

If $\omega$ and $\rho$ solve this problem, then the below decision function $f(x)$ will be positive for most examples $x_{i}$ contained in the training set.

$f(x)=\textit{sign}((\omega.\phi(x))-\rho)$

4. Experimental results

In this section, we compare the proposed models with existing mechanisms for feature set selection, text representation and classifier algorithm, using a public Email dataset. Then the proposed models are evaluated against all controlling parameters to understand their effects on performance and then finally we try to optimize the model.

4.1 Dataset and evaluation parameters

Before proceeding on to the description and evaluation of the proposed models, we will provide a brief overview of the dataset used and the parameters that will be used throughout this paper to evaluate the performance of the models.

4.1.1 Data collection

Emails are the documents of interest chosen for our case study. The Enron public email dataset has been used to experimentally evaluate the techniques suggested in this paper. The Enron corpus consists of a large set of email messages which was made public during the legal investigation concerning the Enron Corporation. It consists of the email records by 158 users, mostly senior management of Enron, organized into folders. This corpus is the only publicly available mass collection of real emails that can be used freely for study.

We have used the email messages from the “sent items” folders of the users for this study. The emails ranged over both personal and business communications. The messages were first filtered to discard emails where there was no contribution from the author, such as forward emails and also discarded contents of the message which was not directly contributed by the author. This filtration step reduced the size of the dataset significantly and resulted in 80 users with at least tens of email messages. Out of these, 50 users folders which had a minimum 200 messages were selected for this experimental analysis. Every experiment under this section was conducted over all these 50 users dataset. However for the sake of clarity, except for the comprehensive performance result (Fig. 5), results for only 10 selected users dataset have been illustrated.

4.1.2 Evaluation measures

To evaluate the result, we use standard measures for performance validation in classification problem. A True Positive (TP) is the state when a positive sample from the trained author’s set gets reported as positive. False Negative state (FN) is when a positive sample of the author gets classified as a negative sample. While testing the classifier with a sample message from any user other than the trained author, if it is classified as positive, then it results in False Positive state (FP). If the email gets classified as negative, then it falls to True Negative (TN) state. Based on these 4 states, we calculate the following parameters that well define the model performance.

Sensitivity/True Positive Rate (TPR): Sensitivity defined as $\textit{TP}/(\textit{TP}+\textit{FN})$ relates to the model’s ability to identify the positive cases correctly. In the scope of our case study, it shows the ratio of the author’s emails that were classified as his/her email correctly. The higher the sensitivity, higher is the probability that the model would classify an authentic (author’s) email as authentic.

Specificity/True Negative Rate (TNR): Specificity defined as $\textit{TN}/(\textit{TN}+\textit{FP})$ relates to the model’s ability to identify the negative cases correctly. It shows the ratio of the negative emails or in other words, the emails that were not written by the author, that were accurately identified as negative samples. Higher the specificity better is the model to detect abnormal cases.

Accuracy: It is the proportion of correctly classified cases among the total samples tested. Higher the accuracy, better the model in identifying the true cases, be it identifying author’s original emails as original or identifying the illegitimate case correctly. But note that accuracy could be misleading when either of TP or TN is dominating.

F ${}_{1}$ score: In order to address the shortcomings of Accuracy, another accuracy parameter called $F_{1}$ score is used. If precision ( $\rho$ ) is the fraction of the true positive instances over the total positive cases reported and recall (r; also called sensitivity) is the fraction of true positive instances over the total positive instances fed to the system, $F_{1}$ score is the harmonic mean of precision and recall. This score presents the best performance with a value of 1 and the worst performance with a value of 0.

$F_{1}∼{}\textit{score}=\frac{\textit{2Precision}\cdot\textit{Recall}}{\textit{% Precision}+\textit{Recall}}$

ROC curve: This Receiver Operating Characteristic (ROC) curve is drawn by plotting True Positive Rate (TPR) to False Positive Rate (FPR), where FPR is calculated as (1-TNR). FPR forms the $x$ -axis and TPR forms the $y$ -axis and the graph is plotted at various threshold levels. The best model would yield 100% specificity and 100% sensitivity and hence the ideal point is the top left corner (0, 1) of the graph. In general, more the area under the curve better is the model.

4.2 Experiment setup

4.2.1 Implementation

The experiments were performed using a standalone java program. A neo4j graph database [26] was used to store the comprehensive graph for each user. The java library libsvm [27] was used to implement one-class SVM. The experiment was performed with values of $\nu$ ranging from 0.0 to 1.0 with a step of 0.1, where the model parameter $\nu$ determines how tightly or loosely the one-class SVM model would fit the training data points. Another open source library encog [28] was used to implement the autoencoder algorithm.

4.2.2 Cross validation

All the experiments related to comparison and evaluation has been performed using 3-fold cross validation. Each user’s email messages have been divided into 3 parts. When the first and second parts were used to train the classifier, the third part was used as the positive test set. Test emails of the same size as that of the positive test set was randomly chosen from the remaining 49 users for negative test set.

4.3 Comparison with other techniques

4.3.1 Feature set selection

We compared the graph-model based feature set selection mechanism with other techniques such as frequency method and TF-IDF method. For implementing Frequency method, a comprehensive frequency-count vector was maintained, which was populated with the frequency of all the terms in the sample space of training set. In order to discard the least frequent terms, the vector was sorted to filter out the terms whose occurrence is less than the first quartile of the frequency distribution. Two schemes of text representation – Frequency representation and ICPM representation were used to analyze this feature set selection technique. Words were considered as terms under frequency representation. Whereas, nodes and links of the comprehensive graph were used as terms in ICPM representation.

For TF-IDF method, frequency of each term ( $t f$ ) was stored in a vector. IDF of each term t was calculated using the Eq. (1), where $N$ is the total number of documents in the sample space and $df_{t}$ is the number of documents containing the term $t$ . TF-IDF vector was obtained by component-wise multiplication of both the vectors. As in case of the previous method, the vector was sorted and feature set filtered out. In this case, we used TF-IDF and ICPM text representation schemes for analysis.

For Graph-model based method, a neo4j database was used in the backend of a java program to store all the vertices and links. Top 2.5% of vertices and 1% of links have been selected as feature set. Both frequency and ICPM representations were analyzed in this case as well.

All three feature-set selection methods with two types of text representation, totally 6 combinations, were tested along with the one-class SVM classifier corresponding to the model parameter $\nu$ value 0.4 and the results are listed in Table 1. Comparison of the parameters sensitivity (TPR), specificity (SPC) and accuracy (ACC) are listed in the table. Average sensitivity of all the models range between 0.5 and 0.6, with an exception of TF-IDF TR scheme. However, the Graph FS-ICPM combination tops the list with a sensitivity of 0.591. Specificity of Graph FS-ICPM TR combination is almost perfect and it is much ahead of all other models. Accuracy of this combination is again notably higher compared to other methods. Interestingly, both the frequency and TF-IDF methods fare better in combination with ICPM TR, than the old TR schemes.

Table 1
Comparison of feature set selection methods

User	Frequency FS						TFIDF FS						Graph FS
	Frequency TR			ICPM TR			TRIDF TR			ICPM TR			Frequency TR			ICPM TR
	TPR	SPC	ACC	TPR	SPC	ACC	TPR	SPC	ACC	TPR	SPC	ACC	TPR	SPC	ACC	TPR	SPC	ACC
User1	0.56	0.72	0.64	0.56	0.60	0.58	0.44	0.63	0.54	0.57	0.50	0.53	0.55	0.65	0.60	0.58	0.98	0.78
User2	0.58	0.75	0.67	0.59	1	0.79	0.52	0.50	0.51	0.59	1	0.79	0.56	0.78	0.67	0.59	1	0.79
User3	0.55	0.55	0.55	0.56	0.63	0.60	0.43	0.61	0.52	0.53	0.89	0.71	0.58	0.63	0.60	0.59	1	0.79
User4	0.54	0.55	0.55	0.51	0.83	0.67	0.42	0.62	0.52	0.55	0.98	0.76	0.54	0.48	0.51	0.59	1	0.79
User5	0.58	0.74	0.66	0.58	0.98	0.78	0.47	0.76	0.62	0.58	0.99	0.78	0.58	0.59	0.58	0.58	1	0.79
User6	0.55	0.59	0.57	0.55	0.43	0.49	0.43	0.68	0.55	0.54	0.39	0.46	0.57	0.57	0.57	0.59	1	0.79
User7	0.58	0.58	0.58	0.55	0.86	0.71	0.50	0.64	0.57	0.57	0.99	0.78	0.58	0.57	0.58	0.59	1	0.79
User8	0.58	0.60	0.59	0.58	0.33	0.45	0.53	0.38	0.45	0.58	0.37	0.47	0.58	0.57	0.58	0.59	0.99	0.79
User9	0.59	0.70	0.64	0.58	0.48	0.53	0.53	0.55	0.54	0.59	0.83	0.71	0.59	0.65	0.62	0.64	0.99	0.81
User10	0.52	0.64	0.58	0.50	0.45	0.47	0.40	0.62	0.51	0.53	0.80	0.67	0.54	0.57	0.56	0.57	1	0.78
Average	0.56	0.64	0.60	0.56	0.66	0.61	0.47	0.60	0.53	0.56	0.77	0.67	0.57	0.61	0.59	0.59	10.00	0.79

Figure 3.

Comparison of feature set selection methods in terms of $F_{1}$ score (left) and ROC curves (right).

$F_{1}$ score and ROC curves of these techniques are compared in Fig. 3. Graph FS-ICPM TR combination could achieve good $F_{1}$ score and also the maximum area under ROC curve. As noted from the evaluation table, the other FS techniques in combination with ICOM TR fare much better than the old TR schemes, with respect to $F_{1}$ score and ROC curve. In summary, the graph model based feature-set selection technique shows better performance compared to other methods.

4.3.2 Text representation

Four text representation schemes are compared with both the proposed schemes PM and ICPM. Since it is already proved that Graph-model based feature selection outperforms all other feature selection techniques, we have based this Text Representation experiment on this optimal feature set selection technique. Also the classifier has been chosen to be the one-class SVM at the $\nu$ value 0.4, as it has been experimentally determined to be the most balanced setting to get optimal performance.

For Binary representation, feature vector of each document was populated with 1 if the corresponding feature was present in the document, else it was marked as 0. In Frequency representation, feature vector was populated with the number of occurrences of each feature in the document. TFIDF representation was implemented by maintaining the feature vector with product of TF and IDF values of each feature, where IDF was calculated using Eq. (1). For Hadamard representation, a frequency vector $\vec{e}$ was constructed for each document. A comprehensive frequency vector $\vec{E}$ for the complete training set was also constructed. Then the feature vector for each document was computed using Eq. (2).

PM and ICPM representations were implemented by creating feature vectors as per Eqs (3), (4) and (5). Performance comparison of all these text representation schemes is listed in Table 2. Sensitivity remains in the range of 0.5-0.6 for all of the representations, with ICPM topping the list. With respect to accuracy, hadamard representation closely follows ICPM. However, specificity of ICPM is almost perfect, and hadamard representation could achieve 0.87, which is much better than other schemes.

Table 2
Comparison of text representation methods

User	Binary TR			Frequency TR			Hadamard TR			PM TR			ICPM TR
	TPR	SPC	ACC	TPR	SPC	ACC	TPR	SPC	ACC	TPR	SPC	ACC	TPR	SPC	ACC
User1	0.56	0.70	0.63	0.55	0.65	0.60	0.60	0.58	0.59	0.58	0.28	0.43	0.58	0.98	0.78
User2	0.57	0.86	0.72	0.56	0.78	0.67	0.58	0.97	0.77	0.60	0.52	0.56	0.59	1	0.79
User3	0.55	0.81	0.68	0.58	0.63	0.60	0.61	0.83	0.72	0.59	0.46	0.53	0.59	1	0.79
User4	0.55	0.68	0.62	0.54	0.48	0.51	0.59	0.82	0.71	0.56	0.55	0.55	0.59	1	0.79
User5	0.58	0.65	0.61	0.58	0.59	0.58	0.56	0.90	0.73	0.59	0.33	0.46	0.58	1	0.79
User6	0.57	0.72	0.65	0.57	0.57	0.57	0.60	1	0.80	0.59	0.42	0.51	0.59	1	0.79
User7	0.57	0.69	0.63	0.58	0.57	0.58	0.60	0.72	0.66	0.60	0.57	0.58	0.59	1	0.79
User8	0.58	0.64	0.61	0.58	0.57	0.58	0.59	0.89	0.74	0.60	0.55	0.58	0.59	0.99	0.79
User9	0.61	0.74	0.67	0.59	0.65	0.62	0.59	0.98	0.78	0.58	0.50	0.54	0.64	0.99	0.81
User10	0.53	0.69	0.61	0.54	0.57	0.56	0.55	1	0.77	0.59	0.37	0.48	0.57	1	0.78
Average	0.567	0.718	0.643	0.567	0.606	0.587	0.587	0.869	0.727	0.588	0.455	0.522	0.591	0.996	0.79

Figure 4.

Comparison of text representation schemes in terms of $F_{1}$ score (left) and ROC curves (right).

$F_{1}$ score and ROC curves are shown in Fig. 4. In these metrics as well ICPM proves to be the most efficient scheme, followed by hadamard representation. Another important observation is that, though PM representation also follows a probabilistic model, it performs well below all the other representations. It can be concluded from all the evaluation parameters that ICPM could represent text more efficiently compared to all other representations.

4.3.3 Classification models

The proposed classification method of graph model based feature selection in combination with ICPM text representation applied on a one class SVM classifier was compared with other classification models in this experiment.

The prototype algorithm was implemented by constructing the tf-idf vector $\vec{d}$ for each document, using Eq. (1). The prototype vector $\vec{c}$ for the class C was constructed with the documents in training set using Eq. (6). Distance of each document $d^{\prime}$ in the sample space to the prototype vector $\vec{c}$ was calculated and the maximum distance was stored as $\delta$ , as per Eq. (7). Any test document that had a distance less than $\lambda\delta$ was classified as a member of the known class. The optimal value for ( $0\leqslant\lambda\leqslant 1$ ) was determined experimentally.

$\delta=\textit{arg}\max\frac{\vec{d^{\prime}}.\vec{c}}{\lVert{\vec{d^{\prime}}% }\rVert.\lVert{\vec{c}}\rVert}$ (7)

Table 3

Comparison of classifier models

User	Prototype			Nearest neighbor			Naïve Bayes			AutoEncoder			One-class SVM
	TPR	SPC	F1	TPR	SPC	F1	TPR	SPC	F1	TPR	SPC	F1	TPR	SPC	F1
User1	0.27	0.62	0.5	0.38	0.41	0.4	0.4	0.46	0.44	0.69	0.12	0.53	0.58	0.98	0.98
User2	0.39	0.64	0.58	0.22	0.43	0.34	0.42	0.52	0.49	0.84	0.58	0.74	0.59	1	1
User3	0.5	0.5	0.5	0.31	0.48	0.42	0.66	0.35	0.41	0.75	0.14	0.57	0.59	1	1
User4	0.35	0.68	0.59	0.53	0.39	0.43	0.45	0.48	0.47	0.82	0.1	0.6	0.59	1	1
User5	0.76	0.21	0.29	0.4	0.41	0.41	0.81	0.11	0.18	0.62	0.26	0.53	0.58	1	1
User6	0.31	0.62	0.52	0.34	0.4	0.38	0.39	0.49	0.46	0.78	0.5	0.69	0.59	1	1
User7	0.32	0.65	0.55	0.38	0.37	0.37	0.51	0.16	0.23	0.49	0.15	0.42	0.59	1	1
User8	0.52	0.59	0.58	0.46	0.44	0.45	0.51	0.54	0.54	0.57	0.57	0.57	0.59	0.99	0.99
User9	0.42	0.68	0.62	0.29	0.46	0.4	0.49	0.5	0.49	0.31	0.23	0.3	0.64	0.99	0.99
User10	0.28	0.70	0.58	0.51	0.48	0.48	0.31	0.59	0.50	0.83	0.78	0.81	0.57	1	1
Average	0.412	0.589	0.531	0.382	0.427	0.408	0.495	0.42	0.421	0.67	0.343	0.576	0.59	0.996	0.996

The nearest neighbor algorithm was implemented by calculating the distance from each document in the sample set to every other document in that sample space. The distance $\delta$ was determined as the maximum distance between any document in the sample space to any other document in the same space. Any test document that had a distance less than $\lambda\delta$ to any other document in the training set was classified as a member of the class. The optimal value for $(0\leqslant\lambda\leqslant 1)$ was determined experimentally.

Figure 5.

Performance of Inclusive Compound Probability Model: Heat-maps corresponding to $\nu$ values 0.2, 0.4, 0.6 and 0.8. Top right corner of a heat-map is the optimal region to hold all the 50 models; Ideal specificity could be gained for 7 models with sensitivity in the range of 0.6–0.7 for $\nu$ -0.2. However, at $\nu$ -0.4, with a little drop in sensitivity (by 0.1), ideal specificity could be gained for 20 models.

Figure 6.

Effect of nu on performance.

Figure 7.

Effect of feature size on specificity.

For Naïve Bayes algorithm, probability of a document $d$ to be member of class $E$ was computed as the product of $p(f|E)$ of all features $f$ in the document, as in Eq. (8). Each $p(f|E)$ was estimated from the frequency $f$ of the feature in the sample space ( $n_{f}$ ), the total number of occurrences of all the features in the sample space $(n)$ and the number of features ( $|dictionary|$ ) using the Eq. (9). Any test document d’ was considered to be member of the class if $p(d^{\prime}|E)$ was greater than or equal to $\lambda\delta$ . $\delta$ was the threshold calculated as the minimum $p(d|E)$ , for all $d\in{E}$ . $(0\leqslant\lambda\leqslant 1)$ was determined experimentally, to find the optimal threshold for acceptance.

$\displaystyle p(d|E)=\prod_{i=1}^{m}{(p(f_{i}|E))^{n_{i}(d)}}$ (8) $\displaystyle p(f|E)=\frac{n_{f}+1}{n+|\textit{dictionary}|}$ (9)

AutoEncoder was implemented with a 3 layer feed forward network. To improve the efficiency, feature set was selected from the top 1% of nodes and 0.5% of links, by discarding features with more than 90% of zeros – in other words the features that had presence in less than 10% of documents were discarded. For all epochs till 200, discarding the initial few epochs for stabilization, average square error at each epoch was calculated and recorded for all the documents in the sample space, to come up with the acceptable level of error (threshold), by discarding the highest 25 percentile error cases from training set.

For testing the proposed Graph FS $+$ ICPM TR $+$ One-class SVM model, we have used the one-class SVM implementation of libsvm [27] with a sigmoid kernel type. The parameter $\nu$ determines how tightly or loosely the model would fit the training data points. For this comparison experiment, the $\nu$ value was set at 0.4, which was experimentally determined to be a balanced position between specificity and sensitivity. More on the effect of $\nu$ will be covered in evaluation section.

Figure 8.

Effect of feature size on sensitivity.

Figure 9.

Optimization of user models.

Table 3 compares the performance of all the above mentioned classification algorithms. The controlling parameter $\lambda$ for prototype, nearest neighbor, Naïve Bayes algorithms, the threshold value in autoencoder and the $\nu$ value in one-class SVM were all fixed at a level to achieve a sensitivity of the range 0.4–0.6. With this pre-fixed sensitivity as base, the specificity and F ${}_{1}$ score of all the classifiers were compared. One class SVM could reach a specificity of 0.99, as well as a F ${}_{1}$ score of 0.99. Rest of all the algorithms performed poorly with much lesser specificity and F ${}_{1}$ . With respect to specificity, prototype algorithm could gain a specificity of above 0.5 among the remaining algorithms. F ${}_{1}$ score of both prototype and autoencoder were above. In conclusion, performance of the proposed model was comparatively higher to other classifiers.

4.4 Evaluation of the classification model

Under this section, we would evaluate the performance of the proposed classification model and analyze the effect of the controlling parameters – nu( $\nu$ ) and feature size over performance. This analysis would be used in the next section to arrive at optimal values for optimal performance.

Figure 5 gives an overview of the performance of the model through heatmaps. Each map corresponds to a particular nu ( $\nu$ ) and shows the distribution of the 50 user models over specificity and sensitivity. An ideal model is the one which could gain a specificity of 1.0 and a sensitivity of 1.0. i.e., it should occupy the top right corner of the map. However, as there is a trade-off between specificity and sensitivity, achieving the ideal position of top tight corner for all the user models is not possible. With less $\nu$ value (0.2), the models move towards right, but their specificity reduces and they drop to the bottom of the graph. On the contrary, with a bigger value of $\nu$ , the models move up but also to the left. An attempt to strike a good balance between these two parameters was performed and at $\nu$ value of 0.4, more than 30 models could be pushed towards the top right position ( ${0.5}\leqslant{sensitivity}\leqslant{1.0};{0.5}\leqslant{specificity}\leqslant% {1.0}$ ).

Effect of $\nu$ .

The parameter $\nu$ of SVM determines how loosely or tightly the model fits the training data. The value of $\nu$ is between 0 and 1, which controls the ratio of support vectors to data points. When $\nu$ is small, say 0.1, the number of support vectors will be a small percentage of the number of data points and hence most part of the training data would not fit in the margin.

When $\nu$ is high, say 0.9, the margin becomes very large and most of the points will fall in the margin. So, at $\nu$ value 0, the model becomes more sensitive and less specific. As the value of $\nu$ moves towards 1.0, the model becomes more specific and less sensitive. This could be clearly seen from the Fig. 6. For certain users, specificity of 1.0 could be achieved at $\nu$ value 0.2 itself. For some users, it is 0.4. When $\nu$ gets closer to 1.0, the model gets tightly fit and all the test samples get classified as negative samples.

Effect of feature size.

The number of features selected to uniquely represent a user style is called the feature size. In graph model, we define feature size as the percentage of top nodes ( $n\\$ ) and top links ( $m\\$ ) that are determined as feature set. It is always recommended to have a smaller feature size, so that the model will not overfit the data and could operate faster.

Figure 7 shows the effect of feature size on specificity. With increase in feature size, the specificity increases. However at a feature size 1% of nodes and 0.5% of links, the model stabilizes and the specificity almost remains constant after that. Figure 8 shows the effect of feature size on sensitivity. It can be seen that for lower feature sizes below 0.25% of nodes and 0.1% of links, the sensitivity oscillates a lot and is not stable. Here again, it could be observed that a feature size more than 1% of nodes and 0.5% of links gives an almost stable sensitivity.

4.5 Optimization

As it was seen from the previous results, there is a trade-off between specificity and sensitivity. In order to get the ideal specificity, the sensitivity needs to be reduced to the range of 0.5–0.6. Though this ratio holds good for all the user models in general, we will be able to reach even better performance for each user model, if they are individually evaluated. Figure 9 shows the evaluation of four user models as an example case study. It could be noted that for User1, with a feature size of 1% of nodes and 0.5% of links at a $\nu$ value of 0.4, the performance reaches 1.0 specificity and 0.6 sensitivity. However for User2, we could achieve a better performance of 1.0 specificity and $>$ 0.8 sensitivity at $\nu$ value 0.1 for a feature size of 1% nodes and 0.5% links itself. Similarly, for User3 also, the ideal specificity with 0.8 sensitivity could be achieved at $\nu$ value 0.2 for a feature size of 1:0.5. But if we check User4, the best performance could be reached at $\nu$ value of 0.2, only for a feature size greater than 5% of nodes and 10% of links.

So, it can be concluded that based on the flexibility to handle larger feature size, performance could be improved. For the best user models, with very less feature size of 1:0.5, we could gain the best performance at lower $\nu$ values.

5. Discussion and conclusion

In the past decade, with the advances in technology, electronic communication has become the main mode of correspondence, which has resulted in extensive usage of short texts such as email, tweets, etc, for conveying messages. In fact, emails are used not only for conveying messages but also for other applications such as, official data transfer (electronic documents), highly secured data transfer (password/PIN numbers), private data transfer (personal information) and even to transfer highly sensitive information (company secrets). When a user sends an email from her mailbox, the sender guarantees the authorship of the corresponding email. The receiver can consider this email as significant as a signed letter from the sender. Even in forensic investigations or legal procedures, an email is granted a lot of significance to stand as a valid proof.

With the novelty detection mechanism proposed by this research work, an additional security interface to detect and prevent email impersonation could be provided by the email servers which would be difficult to break, as manipulating someone’s style of writing is quite tedious. A similar kind of intelligence model could also be used in forensic investigations, where the objective is to find whether a particular suspect has written the given malicious email or not.

In conclusion, we have tried to address the shortcomings of novelty detection mechanisms for short texts through a graph-model based feature extraction technique and a novel text representation scheme. The graph-model based feature extraction technique was compared with other feature-set selection techniques and proved to be much efficient than the existing mechanisms for short texts. Also the ICPM text representation technique is proved to perform better than other methods.

Currently the classification model has been tested only on emails. This could very well be extended for even smaller text messages such as tweets or SMSs. The classification model in this work has focused only on the lexical features of text. Analysis of character and syntactic features could provide further insight into the writing style. Stylometric analysis based on the statistics such as average number of paragraphs, average number of sentences, etc, could also help further improve the performance in novelty detection. Some of the future directions for this work would be to combine this model with character/syntactic features analysis and stylometric analysis as well to achieve even better performance.

Footnotes

Acknowledgments

This research was supported by the Basic Science Research Program through the National Research Foundation (NRF) of Korea funded by the Ministry of Science, ICT, and Future Planning (MSIP) (grant number 2014R1A1A3051169 & 2016R1D1A1B03934129) and was partially supported by the Ajou University research fund.

References

Smith

and Fujinaga

, A review of authorship attribution, Technical Report 15 (2008), 2010.

Holmes

D.I.

, The evolution of stylometry in humanities scholarship, Literary and Linguistic Computing 13(3) (1998), 111–117.

Williams

C.B.

, Mendenhall’s studies of word-length distribution in the works of shakespeare and bacon, Biometrika 62(1) (1975), 207–212.

Burrows

J.F.

, Not unles you ask nicely: The interpretative nexus between analysis and information, Literary and Linguistic Computing 7(2) (1992), 91–109.

Qian

Liu

Chen

and Peng

, Tri-training for authorship attribution with limited training data, in: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 2014.

Howedi

and Mohd

, Text classification for authorship attribution using naive bayes classifier with limited training data, Computer Engineering and Intelligent Systems 5(4) (2014), 48–56..

Can

, Authorship attribution using principal component analysis and competitive neural networks, Mathematical and Computational Applications 19(1) (2014), 21–36.

Joachims

, A probabilistic analysis of the rocchio algorithm with tfidf for text categorization, Technical report, DTIC Document, 1996.

Salton

and McGill

M.J.

, Introduction to modern information retrieval, 1983.

10.

Manevitz

and Yousef

, One-class document classification via neural networks, Neurocomputing 70(7) (2007), 1466–1481.

11.

Manevitz

L.M.

and Yousef

, One-class svms for document classification, The Journal of Achine Learning Research 2 (2002), 139–154.

12.

Diederich

Kindermann

Leopold

and Paass

, Authorship attribution with support vector machines, Applied Intelligence 19(1-2) (2003), 109–123.

13.

Zhao

and Zhang

, An email classification model based on rough set theory, in: Active Media Technology, 2005. (AMT 2005), Proceedings of the 2005 International Conference on, IEEE, 2005, pp. 403–408.

14.

Aery

and Chakravarthy

, Emailsift: Email classification based on structure and content, In Data Mining, Fifth IEEE International Conference on, IEEE, 2005, p. 8.

15.

Brutlag

J.D.

and Meek

, Challenges of the email domain for text classification, in: ICML, 2000, pp. 103–110.

16.

De Vel

Anderson

Corney

and Mohay

, Mining e-mail content for author identification forensics, ACM Sigmod Record 30(4) (2001), 55–64.

17.

Corney

de Vel

Anderson

and Mohay

, Gender-preferential text mining of e-mail discourse, in: Computer Security Applications Conference, Proceedings. 18th Annual, IEEE, 2002, pp. 282–289.

18.

Goodman

Hahn

Marella

Ojar

and Westcott

, The use of stylometry for email author identification: a feasibility study, Proc. Student/Faculty Research Day, CSIS, Pace University, White Plains, NY, 2007, pp. 1–7.

19.

Calix

Connors

Levy

Manzar

MCabe

and Westcott

, Stylometry for e-mail author identification and authentication, Proceedings of CSIS Research Day, Pace University, 2008.

20.

Douglas Baker

Hofmann

McCallum

and Yang

, A hierarchical probabilistic model for novelty detection in text, in: Proceedings of International Conference on Machine Learning, Citeseer, 1999.

21.

Japkowicz

Myers

Gluck

et al., A novelty detection approach to classification, in: IJCAI, 1995, pp. 518–523

22.

Thompson

B.B.

Marks

R.J.

Choi

J.J.

El-Sharkawi

M.A.

Huang

M.-Y.

and Bunje

, Implicit learning in autoencoder novelty assessment, in: Neural Networks, 2002. IJCNN’02. Proceedings of the 2002 International Joint Conference on, Vol. 3, IEEE, 2002, pp. 2878–2883.

23.

Cavnar

W.B.

Trenkle

J.M

et al., N-gram-based text categorization, Ann Arbor MI 48113(2) (1994), 161–175.

24.

Schenker

Last

Bunke

and Kandel

, Classification of web documents using graph matching, International Journal of Pattern Recognition and Artificial Intelligence 18(3) (2004), 475–496.

25.

Schölkopf

Platt

J.C.

Shawe-Taylor

Smola

A.J.

and Williamson

R.C.

, Estimating the support of a high-dimensional distribution, Neural Computation 13(7) (2001), 1443–1471.

26.

Webber

, A programmatic introduction to neo4j, in: Proceedings of the 3rd annual conference on Systems, programming, and applications: software for humanity, ACM, 2012, pp. 217–218.

27.

Chang

C.-C.

and Lin

C.-J.

, Libsvm: a library for support vector machines, ACM Transactions on Intelligent Systems and Technology (TIST) 2(3) (2011), 27.

28.

Heaton

, Programming neural networks with encog 2 in java. heaton research, 2010.