Prediction of reading difficulty in Russian academic texts

Abstract

Education policy makers view measuring academic texts readability and profiling classroom textbooks as a primary task of education management aimed at sustaining quality of reading programs. As Russian readability metrics, i.e. “objective” features of texts determining its complexity for readers, are still a research niche, we undertook a comparative analysis of academic texts features exemplified in textbooks on Social Science and examination texts of Russian as a foreign language. Experiments for 7 classifiers and 4 methods of linear regression on Russian Readability corpus demonstrated that ranking textbooks for native speakers is a much more difficult task than ranking examination texts written (or designed) for foreign students. The authors see a possible reason for this in differences between two processes: acquiring a native language on the one hand and learning a foreign language on the other. The results of the current study are extremely relevant in modern Russia which is joining the Bologna Process and needs to provide profiled texts for all types of learners and testees. Based on a qualitative and quantitative analysis of a text, the research offers a guide for education managers to help build consensus on selecting a reading material when educators have differing views.

Keywords

Text readability machine learning Russian academic text text complexity examination tests

1 Introduction

Modern communication as ’the imparting or exchanging of information by speaking, writing, or using some other medium’ (Oxford English Dictionary, 1996) implies either generating or receiving a text, which may be handwritten, printed, electronic or oral. Successful communication in its turn largely depends on whether the amount, content and structure of the quanta of the information sent by its generator in the text and received by the addressee are similar or in an ideal situation is the same. Thus, for the information of any text (written or oral) to be elicited, processed and stored in the recipient’s mind, it is important that the text itself aligns with the cognitive and linguistic abilities of the recipient. Matching a text to the target audience is a problem relevant in a number of spheres: the military, education, PR, advertising, government, business, publishing, medicine and social relations as these are the areas where communication is the foundation of success. The research shows that companies suffer damages and take financial hits if the texts to which they expose their customers are hard for the average reader to read [1]. If a text is too easy, i.e. primitive for the audience, readers lose their interest and stop reading. In modern science the problem of text complexity is positioned within a wider problem of ‘text – reader’ and is correlated with the notions of text readability and text complexity. Text readability/complexity is generally defined as the total amount of factors influencing success of reading by the target audience. The latter in its turn is dependent on the degree of reading comprehension at a certain reading speed and maintaining interest in the content of the text. Modern researchers address the problem of ‘the text – reader correspondence’ in two different ways: (a) from the point of view of the reader and his/her individual characteristics, such as age, education, background information, memory span, etc; and (b) from the point of view of the text and its objective features, such as length and ‘complexity’. In this research aimed at revealing the effectiveness of various algorithms in ranking complexity/readability of academic text we diversified the main features: a set of linguistic features affecting complexity/readability of the text and the size of the text. The best results were obtained for the complete set of text features and the longest of all the analyzed texts. The application of methods of machine learning allowed us to take into account number of features simultaneously while earlier no more than a range of three variables was used to assess text complexity/readability. The results obtained also demonstrate that ranking readability of educational texts is a much more difficult task than ranking texts of tests. The latter is probably caused by the language proficiency, i.e. the range and size of the active vocabulary of the target audience: native language proficiency among pupils of the 5th -11th grades is relatively comparable resulting in readability values of academic texts of grades 5 - 11 being very much close or even similar to each other.

1.1 Linear models for Russian academic texts

The Flesch Reading Ease (FRE), and the Flesch – Kincaid Grade Level (FKG) [2] are two readability tests designed to indicate how difficult is a passage in English for understanding. Based on the differences between Russian and English, discovered by Oborneva in 2006 [3], the researcher adapted the Flesch Reading Ease Readability Formula for Russian texts as follows:

$FRE = 206.836 - (1.52 \times ASL) - (65.14 \times ASW) .$

The formula takes into account two variables: ASL – the average number of words per sentence, and ASW – the average number of syllables per word. The following two modified linear models were obtained based on the corpus of school textbooks on Social Studies [4, 28]. Features of the model are presented below (referred to as Russian Readability Formulas: RRF-1 and RRF-2):

$\begin{matrix} {FRE}_{\mod} (ASL, ASW) = \\ 208.7 - 2.6 \times ASL - 39.2 \times ASW \end{matrix}$ (RRF-1)

$\begin{matrix} {FKG}_{\mod} (ASL, ASW) = \\ 0.36 \times ASL + 5.76 \times ASW - 11.97 \end{matrix}$ (RRF-2)

1.2 Research questions

In the paper we aim at the following research questions (RQ):

RQ.1: Compare and contrast the effectiveness of various algorithms of machine learning, sets of features, sizes of texts on ranking complexity/readability of academic texts;

RQ.2: Compare and contrast the best results obtained for Russian academic texts with the results of various studies aimed at ranking readability of tests of Russian as a foreign language.

2 Related works

Matching a text to the target audience is a problem relevant in a number of spheres: management, the military, education, PR, advertising, government, business, publishing, medicine and social relations as these are the areas where communication is the foundation of success. Text readability as a problem attracts a number of researchers [5 –7], basically for English.

State of the art studies in the field of ranking readability of texts use various methods of machine learning [8 –16].

The task of searching and evaluating the complexity of texts for teaching languages for a wide range of languages was carried out in [17]. In this work, the authors use Principal Component Analysis to select features able to rank readability of texts.

Similarly, a series of works [18 , 21–24] has been devoted to automatic classification of readability/complexity of tests of Russian as a foreign language. Several classification algorithms were considered in [18]: Random Forest, NNge (nearest-neighbor with non-nested generalized exemplars), FT (Functional Trees), Multilayer Perceptron, and SMO (sequential minimal optimization for support vector machine) from WEKA data mining software [19]. Random Forest [20] is a commonly used classification algorithm, the merits of which are well known. It was also successfully used in [21, 22]. Several methods were applied in [24]: Classification Tree, SVM, and Logistic Regression models. One of the methods of linear regression is the Ridge Regression was used in [23]. The best result for the standard F1-measure = 0.67 was obtained in [18, 21]. To the best of our knowledge, no studies have been carried out in the area in which methods of machine learning were used for ranking readability/complexity of academic texts for secondary schools in Russian.

3 Dataset

3.1 Collecting a corpus of school textbooks

We chose to compile a corpus of Social Studies classroom textbooks based on the assumption that their usefulness in the research of this kind is higher than other discipline textbooks as they do not contain charts, maps, pictures and graphs of any kind which may contribute to text readability. Thus, by providing ‘pure texts’ we eliminate possible ‘noise’ of text accompanying elements. As it was mentioned earlier we compiled a corpus of 14 classroom books: 7 textbooks for grades 5-11 edited by Nikitin and 7 textbooks edited by Bogolubov. The sizes of documents are measured in tokens, i.e. the total number of words in a text. Star sign (*) denotes the so-called ‘advanced’ versions of textbooks, i.e. tailored to preferences and interests of more advanced students in the corresponding grades. Students using these books usually target at sitting a matriculation exam (EGE) in the subject. Sign “–” denotes absence of a textbook of a particular author for the corresponding grade in the corpus. The total size of the corpus is presented in Table 1 which contains the number of tokens in each of the documents as well as labels of readability derived in the following way: we manually assigned the corresponding grade of a particular textbook to as a target variable value (‘grade level’).

Table 1
Sizes (in tokens) of documents in the RRC corpus

Grade BOG NIK Label

5-th – 20,788 5

6-th 17,586 17,392 6

7-th 24,255 25,157 7

8-th 53,077 43,395 8

9-th 45,416 47,212 9

10-th 81,974 42,030 10

10-th* 105,840 – 11

11-th – 41,454 11

11-th* 105,735 – 12

Grade	BOG	NIK	Label
5-th	–	20,788	5
6-th	17,586	17,392	6
7-th	24,255	25,157	7
8-th	53,077	43,395	8
9-th	45,416	47,212	9
10-th	81,974	42,030	10
10-th*	105,840	–	11
11-th	–	41,454	11
11-th*	105,735	–	12

For instance, value “8” corresponds to two textbooks (8 BOG, 8 NIK) used as a part of Social Studies course during the 8-th year of studies in Russian secondary schools.

3.2 Pre-processing of the corpus

For convenience, we have preprocessed all texts from the corpus in the same way. Common preprocessing included tokenization, splitting text into sentences and part-of-speech tagging (with the help of the TreeTagger for Russian 1 ). During the preprocessing step we excluded less than 1. The reasons for the latter are stated below. Extremely short sentences appear mostly as names of chapters and sections of the textbooks studied or a result of incorrect sentence splitting. They lack a grammatical structure of a sentence presenting a sequence of words, i.e. word combinations. Moreover sentences with five to seven words are viewed as short sentences in Russian, as the average sentence length (in our corpus) is higher than ten. We also eliminate sentences longer than 120 words as all of them are quotations from Constitution of the Russian Federation and present samples of a different type of discourse. As the average sentence length is a very important feature in text complexity assessment and hence should not be biased by unrepresentative patterns it was decided to exclude both patterns (shorter than 5 and longer than 120) from the Corpus. To ensure reproducibility of results, we uploaded the corpus on the website thus providing its availability online 2 .

3.3 Sampling from the corpus

Modern views on the size of a representative corpus are based on the notion that all text features have definite frequency patterns which differ for different types of texts, genres, and discourse. A corpus becomes representative when its size is ample for all its features to be exemplified with their typical frequency. Researchers have accumulated a certain amount of knowledge on various text features occurrences: e.g. Biber’s studies [25] revealed a relative stability of grammatical features of a text across 1,000-word samples of number of linguistic features in the English language: pronouns, contractions, past and present tense and prepositions. Biber also confirmed that “linguistic tendencies are quite stable with ten (and to some extent even five) text samples per genre or register” [25]. Though it is obvious that text features occur with a definite regularity, thus making their average metrics, these regularities are still research niches for many types of Russian academic texts of different complexity levels. E.g. we do not know average academic metrics for academic texts aimed at children of different academic age. Thus, we compiled a corpus of 14 academic texts, a complete set of texts of textbooks of two authors, Nikitin and Bogolubov on Social Studies, and marked the corpus as RRC (i.e. Russian Readability Corpus). As it was mentioned earlier, RRC contains 14 documents and thus by no means could be treated as a representative sample of the population of the school textbooks under study. Building a larger corpus is difficult, because it would violate some of the key properties: either new texts will come from other domains, or texts will come from different authors with different writing styles. Both cases of this kind may add noise to the existing dataset.

In order to overcome the problem of the limited size of the collection, the following procedure of sampling from the corpus was suggested in [4]. It is assumed, that text complexity measured on a (significant) part of a text should be close to the complexity of the whole text. This assumption means, that starting from a certain subset (or a sample) of sentences, text complexity of the sample is to be almost the same as text complexity of the whole document from a corpus. The actual plots of the complexity metrics are represented in Fig. 1.

Fig.1

FKG_mod values for different number of sentences sampled from a document [4].

3.4 Results of sampling

Using the sampling technique we can generate more texts from the original corpus. To preserve natural sequence of the sentences we split each textbook into consequent parts. Each part of this type is of a fixed sample size (number of tokens), S = 500, 1000, 2000, or 4000. We use a token, not a sentence, as a unit to measure the size of a sample based on the assumption that the sentence length is different in textbooks designed for different grades. All the samples from a particular document are marked with the same grade level which is used as a label for training. As the last step, we remove outliers from the collection of samples. The samples selected have the number of tokens equal to S±10% 3 . The size of a collection depends on the parameter S (Table 2)). Given ‘S’ as a restriction for the length of each sample, we generate a collection of texts balanced in length.

Table 2
Number of samples in a collection of size S (number of tokens)

Label S=500 S=1000 S=2000 S=4000

5 46 23 11 5

6 77 38 18 9

7 106 54 27 13

8 205 106 53 26

9 196 102 51 25

10 263 137 68 33

11 307 162 81 40

12 227 116 58 29

Total 1427 738 367 180

Label	S=500	S=1000	S=2000	S=4000
5	46	23	11	5
6	77	38	18	9
7	106	54	27	13
8	205	106	53	26
9	196	102	51	25
10	263	137	68	33
11	307	162	81	40
12	227	116	58	29
Total	1427	738	367	180

3.5 Features

We have explored the following sets of features for text complexity prediction:

size-based features (Group-1):

average number of words per sentence (ASL),

average number of syllables per word (ASW),

frequency-based features (Group-2):

frequency of content words (FREQ),

frequency of long words (LONGW),

type-token ratio (TTR),

features based on POS-tags (Group-3):

proportion of nouns (NOUNS),

proportion of verbs (VERBS),

proportion of adjectives (ADJ),

proportion of pronouns (PNOUNS),

proportion of words in Nominative case (NOM),

proportion of words in Genitive case (GEN).

In order to calculate ‘FREQ’ we first collected a set of content words T. The set of content words T is defined as a subset of all words from a given text (TextWords) that also appear in the Russian frequency dictionary collected by Sharov and Lyashevskaya [26] (FreqDict), but do not appear in the set of stop words (StopWords).

$T = TextWords \cap FreqDict \ StopWords$

$FREQ = \frac{1}{| T |} \sum_{w \in T} FreqDict (w)$

‘LONGW’ is calculated as a proportion of words with four or more syllables. The feature ‘TTR’ is a proportion of the number of types of words (or unique words) to the number of all tokens in a document. All features from Group-3 were calculated as proportions of words belonging to a corresponding Part-of-Speech. Figure 2 shows the distribution of the features across the samples. Figure shows unnormalized, raw values of the features. Before the training step, the dataset was normalized with the help of the Min-Max scaler.

Fig.2

Values of features in the collection of samples generated from the original corpus; S=4000.

The Russian language, unlike English, has a rich morphology, including cases of nouns, adjectives and pronouns. This is due to the use in the study of morphological features of the proportion of words in the Nominative case (NOM), the proportion of words in the Genitive case (GEN). It is expected that with increasing complexity of texts, the proportion of nominal groups in indirect cases (not nominative) will increase, of which the most frequent is the genitive case. The obvious difference from the studies for the English language is the use of various external resources, such as frequency dictionaries.

4 Regression and classification in text readability prediction

We can address the problem of text complexity prediction from two points of view depending on how we treat the target variable (the grade level). If the grade level is a real number, then its value can be predicted using a regression model. Otherwise, if one treats the ‘grade level’ as a discrete categorical variable (a variable with values ‘5th grade’, ‘6th grade’, ‘7th grade’, etc.), then we can train a classification model for prediction of the correct class label.

4.1 Methodology for evaluation of readability prediction

We run all evaluation using implementation of the machine learning methods from the Scikit-Learn library with default settings. In our experiments we use well-known performance measures: MSE and MAE for regression, and F1-score with macro-averaging for classification performance. In addition, we provide confusion matrices for classification results of the classifiers that perform better than others. During evaluation of performance measures we use 10-fold cross-validation. Finally, we provide learning curves for classifiers to show how the estimator works for different size of training set.

In the experiments with classifiers we use the following models from the Scikit-Learn library with default settings.

Decision Tree classifier (DT)

Multi-layer Perceptron classifier (MLP)

Random Forest classifier (RF)

k-Nearest Neighbors classifier (kNN)

Support Vector classifier (SVC)

SGD classifier (SGD)

Gradient Boosting classifier (GBoost)

In experiments with regression we use the following models from the Scikit-Learn library:

Linear Regression (LR)

Linear Regression with Ridge Regularization (Ridge)

Linear Regression with Lasso Regularization (Lasso)

Linear Regression with ElasticNet Regularization (ElasticNet)

4.2 Results of classification

Results of complexity prediction using the classifiers are presented in Tables 4–6. Additionally, we evaluated performance of binary classification in the experiments with two pairs of classes: (1) ‘5th grade’ versus ‘8th grade’ and (2) ‘9th grade’ versus ‘12th grade’ (Table 3). The learning curve for the Random Forest classifier is presented in Figure 4. The curve represents a typical situation when a classifier has not enough data to perform better. As one can see starting from 400 training instances F1-measure approaches 0.4. At the same time on the test set this classifier performs well. It means that additional (and possible more diverse) portion data would improve the performance.

Table 3
F1-measure derived using all features; S=4000

Classifier ‘5th ’ vs. ‘8th’ ‘9th’ vs. ‘12th’

DT 1.00 (±0.00) 0.85 (±0.33)

MLP 0.94 (±0.34) 0.92 (±0.19)

RF 0.94 (±0.34) 0.87 (±0.24)

kNN 0.94 (±0.34) 0.92 (±0.19)

SVC 0.71 (±0.57) 0.92 (±0.19)

SGD 1.00 (±0.00) 0.85 (±0.43)

Gboost 0.94 (±0.34) 0.82 (±0.41)

Classifier	‘5th ’ vs. ‘8th’	‘9th’ vs. ‘12th’
DT	1.00	(±0.00)	0.85	(±0.33)
MLP	0.94	(±0.34)	0.92	(±0.19)
RF	0.94	(±0.34)	0.87	(±0.24)
kNN	0.94	(±0.34)	0.92	(±0.19)
SVC	0.71	(±0.57)	0.92	(±0.19)
SGD	1.00	(±0.00)	0.85	(±0.43)
Gboost	0.94	(±0.34)	0.82	(±0.41)

Table 4

F1-measure derived using all features

Classifier	S=500	S=1000	S=2000	S=4000
DT	0.30 (± 0.09)	0.36 (± 0.13)	0.35 (± 0.14)	0.35 (± 0.20)
MLP	0.34 (± 0.09)	0.37 (± 0.09)	0.40 (± 0.11)	0.34 (± 0.18)
RF	0.26 (± 0.06)	0.25 (± 0.09)	0.33 (± 0.14)	0.28 (± 0.24)
kNN	0.34 (± 0.12)	0.36 (± 0.10)	0.39 (± 0.12)	0.47 (± 0.21)
SVC	0.22 (± 0.07)	0.18 (± 0.11)	0.15 (± 0.06)	0.08 (± 0.10)
SGD	0.17 (± 0.14)	0.20 (± 0.09)	0.17 (± 0.16)	0.22 (± 0.18)
Gboost	0.33 (± 0.06)	0.35 (± 0.12)	0.34 (± 0.17)	0.41 (± 0.16)

Fig.3

Confusion Matrix of the Random Forest Classifier.

Fig.4

Learning curve depicts how performance of the Random Forest Classifier depends on the size of a train set. Dashed line represents performance on the train set, solid line represents performance on test set.

Table 5

F1-measure derived using features from Group-1

Classifier	S=500	S=1000	S=2000	S=4000
DT	0.28 (± 0.06)	0.30 (± 0.12)	0.33 (± 0.11)	0.32 (± 0.14)
MLP	0.29 (± 0.06)	0.31 (± 0.10)	0.20 (± 0.11)	0.13 (± 0.10)
RF	0.25 (± 0.09)	0.28 (± 0.07)	0.34 (± 0.19)	0.30 (± 0.09)
kNN	0.29 (± 0.05)	0.30 (± 0.06)	0.34 (± 0.16)	0.37 (± 0.16)
SVC	0.24 (± 0.06)	0.24 (± 0.08)	0.16 (± 0.14)	0.11 (± 0.10)
SGD	0.16 (± 0.06)	0.16 (± 0.12)	0.20 (± 0.16)	0.17 (± 0.16)
Gboost	0.31 (± 0.08)	0.31 (± 0.11)	0.37 (± 0.11)	0.40 (± 0.22)

Table 6

F1-measure derived using features from Group-2 and Group-3

Classifier	S=500	S=1000	S=2000	S=4000
DT	0.28 (± 0.05)	0.29 (± 0.16)	0.32 (± 0.23)	0.21 (± 0.15)
MLP	0.31 (± 0.11)	0.36 (± 0.08)	0.34 (± 0.20)	0.22 (± 0.11)
RF	0.22 (± 0.08)	0.24 (± 0.11)	0.29 (± 0.16)	0.22 (± 0.10)
kNN	0.29 (± 0.08)	0.34 (± 0.11)	0.35 (± 0.18)	0.27 (± 0.12)
SVC	0.19 (± 0.08)	0.17 (± 0.10)	0.15 (± 0.06)	0.09 (± 0.08)
SGD	0.16 (± 0.12)	0.17 (± 0.10)	0.20 (± 0.15)	0.15 (± 0.14)
Gboost	0.30 (± 0.08)	0.34 (± 0.13)	0.35 (± 0.21)	0.24 (± 0.14)

The results of complexity/readability prediction done with the help of the regression models are presented in Table 7 from the Appendix.

Table 7

MSE and MAE derived using all features

	S=500		S=1000		S=2000		S=4000
Model	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
	All features
LR	1.75 ±0.32	1.08 ±0.11	1.47 ±0.42	0.99 ±0.16	1.23 ±0.51	0.90 ±0.21	1.04 ±0.57	0.83 ±0.26
Ridge	1.76 ±0.32	1.08 ±0.11	1.47 ±0.38	0.99 ±0.14	1.23 ±0.54	0.90 ±0.21	1.03 ±0.52	0.83 ±0.24
Lasso	1.81 ±0.35	1.10 ±0.12	1.53 ±0.47	1.00 ±0.18	1.27 ±0.54	0.91 ±0.23	1.07 ±0.62	0.85 ±0.26
ElasticNet	1.83 ±0.37	1.11 ±0.13	1.54 ±0.40	1.01 ±0.16	1.29 ±0.52	0.92 ±0.21	1.06 ±0.63	0.85 ±0.25
	Group-1
LR	1.86 ±0.34	1.11 ±0.12	1.56 ±0.48	1.01 ±0.17	1.28 ±0.58	0.91 ±0.24	1.08 ±0.71	0.84 ±0.29
Ridge	1.86 ±0.44	1.11 ±0.14	1.56 ±0.45	1.01 ±0.16	1.29 ±0.51	0.91 ±0.21	1.08 ±0.65	0.84 ±0.26
Lasso	1.86 ±0.42	1.12 ±0.14	1.57 ±0.39	1.01 ±0.15	1.29 ±0.53	0.92 ±0.20	1.08 ±0.69	0.84 ±0.27
ElasticNet	1.89 ±0.33	1.13 ±0.11	1.60 ±0.41	1.03 ±0.15	1.32 ±0.53	0.93 ±0.22	1.10 ±0.68	0.84 ±0.28
	Group-1 and Group-2
LR	1.85 ±0.39	1.10 ±0.13	1.56 ±0.49	1.00 ±0.18	1.30 ±0.51	0.91 ±0.19	1.11 ±0.71	0.84 ±0.28
Ridge	1.85 ±0.42	1.10 ±0.14	1.56 ±0.50	1.00 ±0.19	1.30 ±0.51	0.91 ±0.21	1.11 ±0.63	0.84 ±0.25
Lasso	1.86 ±0.39	1.11 ±0.14	1.57 ±0.40	1.01 ±0.15	1.30 ±0.52	0.92 ±0.22	1.09 ±0.69	0.84 ±0.27
ElasticNet	1.87 ±0.31	1.12 ±0.12	1.58 ±0.42	1.02 ±0.15	1.31 ±0.52	0.92 ±0.21	1.10 ±0.63	0.85 ±0.26
	Group-1 and Group-3
LR	1.75 ±0.33	1.08 ±0.11	1.47 ±0.40	0.98 ±0.14	1.21 ±0.46	0.89 ±0.20	1.01 ±0.53	0.83 ±0.26
Ridge	1.75 ±0.35	1.08 ±0.12	1.47 ±0.40	0.99 ±0.15	1.21 ±0.43	0.89 ±0.18	1.01 ±0.53	0.83 ±0.25
Lasso	1.81 ±0.34	1.10 ±0.12	1.52 ±0.41	1.00 ±0.17	1.27 ±0.51	0.91 ±0.21	1.04 ±0.62	0.85 ±0.28
ElasticNet	1.86 ±0.34	1.12 ±0.12	1.57 ±0.40	1.03 ±0.16	1.30 ±0.50	0.93 ±0.20	1.07 ±0.58	0.87 ±0.24
	Group-2 and Group-3
LR	1.90 ±0.36	1.13 ±0.12	1.60 ±0.38	1.04 ±0.14	1.35 ±0.50	0.94 ±0.20	1.16 ±0.60	0.88 ±0.26
Ridge	1.90 ±0.40	1.13 ±0.13	1.60 ±0.40	1.04 ±0.15	1.35 ±0.52	0.94 ±0.21	1.15 ±0.73	0.88 ±0.30
Lasso	1.97 ±0.38	1.16 ±0.12	1.67 ±0.48	1.06 ±0.17	1.43 ±0.61	0.98 ±0.23	1.21 ±0.68	0.91 ±0.27
ElasticNet	2.00 ±0.42	1.18 ±0.13	1.71 ±0.45	1.08 ±0.17	1.46 ±0.58	0.99 ±0.22	1.24 ±0.75	0.92 ±0.28

5 Discussion

As we see, although we applied the same research methodologies for academic texts on the one hand and examination texts on the other, our results for school textbooks differ considerably: the best result obtained for F1 on academic texts equals 0.47, while the value for Russian examination texts is 0.67 [18, 21]. The following explanation is viewed by the authors as reasonable: examination texts used in tests of a foreign language is ranked by The Common European Framework of References (CEFR) into 3 main difficulty levels: A, B, C, with A being the easiest and C - close to the proficiency of a native speaker (https://www.examenglish.com/CEFR/cefr.php). Each of the levels is also subdivided into two: A1, A2, B1, B2, C1, C2. Nowadays test developers also make a distinction between the ‘criterion levels’ (e.g. C2) and the ‘plus levels’ (e.g. C2+) thus admitting more subtle divisions of texts complexity. Tests of six CERF levels correspond to significantly different levels of knowledge of the language and, accordingly, vary considerably in complexity.

As for the textbooks under study, they are written for native speakers, who know their native language almost equally, though they go to different grades, from the 5th to the 11th: speech of the 5-graders no longer contains grammatical errors. The main difference in the language proficiency between the 5th and 11th graders is in the sizes of their active and passive lexicons. However, this difference is not considerable. Golovin argues [27], that the average passive vocabulary of a 5-grader is 31 000 words, and that of an 11-grader is 53 000, i.e. only 1.7 times higher.

At the same time, the difference in the size of the vocabulary studied by a speaker of each of the levels described above is much higher: for the English language it is estimated by the British Council 4 , as follows Fig. 5. Thus, a C2 speaker and reader is expected to know 8 times more words than an A1 speaker. Work 5 specifies even higher vocabulary standards for TORFL tests.

Fig.5

Comparison of English and Russian standards for vocabulary size.

According to these data, vocabulary requirements increase 11 times within the span A1 equals C2. Lexical features occupy a significant place among the processed features: e.g. the authors in [21] assessed correlation of lexical features and test complexity levels were calculated and discovered that only 5 features have a Pearson correlation coefficient ≥ 0.7, but four of them are lexical. Moreover, the highest correlation was shown by the lexical minimums coverage [21]. Work [18] convincingly proves that lexical features only ensure the value of F1 equal to 0.639, which is almost the same as for the complete set of features. Thus, we may conclude that the task aimed at in this paper, i.e. automatic ranking of Russian academic texts complexity, is much more complicated than ranking examination texts of Russian as a foreign language. To illustrate this, in Fig. 6 we plot random samples from the corpus (one dot per a 1000-token sample). It is clear that samples from different grade levels plotted close to each other (or “mixed”) if we consider basic features only. The latter implies that textbooks levelling for secondary and high schools require metrics of more specific and elaborated text features. The perspective of the study lies in measuring text cohesion which is a complex metric comprised of lexical and semantic overlaps, argument and stem overlaps, etc.,

Fig.6

A scatter-plot of the collection of samples generated from the original RRC corpus, using size of the sample S=1000 tokens. Different shapes and color represents the label of a text sample. The histograms in main diagonal represent densities of corresponding features.

6 Conclusion

The article presents the results of applying a wide range of machine learning methods to rank complexity/readability of Russian academic texts. The study was carried out on the corpus of textbooks on Social Studies for grades 5 - 11 compiled by the authors for previous studies. The main goal of the research was to compare and contrast the effectiveness of various algorithms of machine learning, sets of features, sizes of texts on ranking complexity/readability of texts in the tests Russian as a foreign language as well as texts of school textbooks. While the first type of texts, i.e. collections of exam or test texts, are among the most frequently selected corpora in readability studies, the second proved to be beyond the area of interest of researchers. In this study we conducted experiments for 7 classifiers and 4 methods of linear regression on the Corpus of Russian textbooks and diversified a set of linguistic features affecting complexity/readability of the text and the size of the text. The choice of the features was based on the results of the previous studies conducted in Russia and abroad. The results obtained in the current study are estimated by standard methods, i.e. F1 measure for classifiers and MSE for linear regression.

The main contribution of the article, i.e. that ranking readability of educational texts is a much more difficult task than ranking texts of tests, is probably true for all languages. If for some language there is a sufficiently large digital collection of textbooks for different classes and a ranked collection of tests of this language as foreign, then the methods proposed in this work (sampling texts, selecting lengths of texts, machine learning, etc.) can be used for another language as well. The considered set of features can be recommended for languages of inflectional type.

Variants of the linear regression demonstrate very similar results, while the results for the classifiers selected vary considerably and the kNN classifier gives the highest scores for measure F1. The best results are obtained for the complete set of features and the longest of all the analyzed texts. The application of methods of machine learning makes it possible to take into account the number of features simultaneously while in all Russian readability formulas no more than a range of three variables was used to assess text complexity/readability. All this dramatically raises the quality of measuring the complexity of texts. The results obtained by the authors of the article demonstrate that ranking readability of educational texts is a much more difficult task than ranking texts of tests. We consider the latter to be caused by the language proficiency of a reader, i.e. the range and size of the active vocabulary of the target audience: native language proficiency among pupils of the 5th -11th grades is relatively comparable resulting in readability values of academic texts of grades 5 - 11 being very much close or even similar to each other. Complexity/readability of texts in tests of Russian as a foreign language, on the contrary, varies greatly in texts aimed for starters (level A1) and those of the proficiency level (C2).

Unfortunately, the estimates of ranking results in both tasks, i.e. for academic texts (textbooks) and tests (examination texts), are far from being high: 0.47 and 0.67, respectively. Apparently, it proves that the methods of using easily computable morphosyntactic, lexical and frequency metrics have exhausted themselves. Better results can probably be achieved with more complex features, such as referential cohesion and deep cohesion as well as information about individual characteristics of a reader,e.g. his/her age, background information of the world, ‘active’ and ‘passive’ vocabulary, etc. We also plan to use more powerful methods of machine learning - the newly developed methods of deep learning of neural networks in our future studies. We also argue that the suggested methods of text complexity measurement, though not unlimited, still provide a solid foundation for profiling academic texts for both L1 and L2 students which eventually contributes to better educational policies, fairness, reliability and validity of tests. Texts which are correctly selected for the target audience of readers motivate students’ success and are better instruments in classroom management.

Footnotes

Obviously, most of the samples collected do not have the exact number of S tokens. We did not include into the collection of samples too short documents, which appear at the end of each document.

Acknowledgments

This research was financially supported by the Russian Science Foundation, grant # 18-18-00436, the Russian Government Program of Competitive Growth of Kazan Federal University, and the subsidy for the state assignment in the sphere of scientific activity, grant agreement 34.5517.2017/6.7. The Russian Academic Corpus (section 3.1, 3.2 in the paper) was created without support from the Russian Science Foundation.

References

Klare

G.R.

, The measurement of readability: Useful information for communicators, ACM J Comput Doc24(3) (2000), 107–121. DOI https://dx-doi-org.web.bisu.edu.cn/10.1145/344599.344630

Flesch

, How to write plain English: A book for lawyers and consumers, Harpercollins, 1979.

Obobroneva

, Avtomatizirovannaya otsenka slozhnosti uchebnykh tekstov na osnove statisticheskikh parametrov [Semiautomatic evaluation of the complexity of academic texts on the base of statistic parameters], Moscow: RAE institute of content and methods of teaching. M.: RAS Institut soderzhaniya i metodov obucheniya, 2006.

Solovyev

, Ivanov

and Solnyshkina

, Assessment of reading difficulty levels in Russian academic texts: Approaches and Metrics, Journal of Intelligent & Fuzzy Systems34(5) (2018), 3049–3058.

Jiang

, Gu

, Yin

and Chen

, Enriching Word Embeddings with Domain Knowledge for Readability Assessment, In Proceedings of the 27th International Conference on Computational Linguistics, 2018, pp. 366–378.

Collins-Thompson

, Computational assessment of text readability: A survey of current and future research, ITLInternational Journal of Applied Linguistics165(2) (2014), 97–135.

Palotti

J.R.D.M.

, Zuccon

and Hanbury

, The influence of pre-processing on the estimation of readability of web documents, In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, 2015, pp. 1763–1766.

Petersen

S.E.

and Ostendorf

, A machine learning approach to reading level assessment, Computer Speech and Language23 (2009), 86–106.

Schwarm

and Ostendorf

, Reading level assessment using support vector machines and statistical language models, In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL-05), 2005, pp. 523–530.

10.

Feng

, Automatic Readability Assessment, PhD thesis, City University of New York (CUNY), 2010.

11.

Feng

, Jansche

, Huenerfauth

and Elhadad

, A comparison of features for automatic readability assessment, In Proceedings of the 23rd International Conference on Computational Linguistics (COLING 2010), 2010.

12.

Graesser

A.C.

, McNamara

D.S.

, Louweerse

M.M.

and Cai

, Cohmetrix: Analysis of text on cohesion and language, Behavior Research Methods, Instruments and Computers36 (2004), 193–202.

13.

Crossley

S.A.

, Dufty

D.F.

, McCarthy

P.M.

and McNamara

D.S.

, Toward a new readability: A mixed model approach, In McNamara

D.S.

and Trafton

, editors, Proceedings of the 29th Annual Conference of the Cognitive Science Society Cognitive Science Society, 2007.

14.

Crossley

S.A.

, Greenfield

and McNamara

D.S.

, Assessing text readability using cognitively based indices, Teachers of English to Speakers of Other Languages (2008), 475–493.

15.

Crossley

S.A.

, Louwerse

M.M.

, McCarthy

P.M.

and Mc-Namara

D.S.

, A linguistic analysis of simplified and authentic texts, The Modern Language Journal, 2007.

16.

Vajjala

, Analyzing Text Complexity and Text Simplification: Connecting Linguistics, Processing and Educational Applications, PhD thesis, University of Tubingen, 2015.

17.

Sharoff

, Kurella

and Hartley

, Seeking needles in the web’s haystack: Finding texts suitable for language learners, In Proceedings of the 8th Teaching and Language Corpora Conference,2008.

18.

Reynolds

, Insights from Russian second language readability classification: Complexity-dependent training requirements, and feature evaluation of multiple categories, In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, 2016, pp. 289–300.

19.

Hall

, Frank

, Holmes

, Pfahringer

, Reutemann

and Witten

I.H.

, The WEKA data mining software: An update, In The SIGKDD Explorations, volume 11, 2009, pp. 10–18.

20.

Breiman

, Random forests, Machine Learning45(1) (2001), 5–32.

21.

Sadov

M.A.

, Razrabotka podhoda dlja izmerenija chitaemosty tekstov na russom jazyke, Magisterskaja rabota, 2018, Vysshaja shkola ekonomiki.

22.

Laposhina

A.N.

, Veselovskaya

T.V.

, Lebedeva

M.U.

and Kupreshchenko

O.F.

, Automated Text Readability Assessment for Russian Second Language Learners, Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference, 18, 2018, pp. 1–11.

23.

Laposhina

, Relevant features selection for the automatic text complexity measurement for Russian as a foreign language, Computational Linguistics and Intellectual Technologies: Papers from the Annual International Conference “Dialogue”, V. 17, 2017, pp. 1–7.

24.

Karpov

, Baranova

and Vitugin

, Single-sentence readability prediction in Russian, In Proceedings of Analysis of Images, Social Networks, and Texts Conference (AIST), 2014, pp. 91–100.

25.

Biber

, Methodological issues regarding corpus-based analyses of linguistic variation, Literary and Linguistic Computing5(4) (1990), 257–269.

26.

Lyashevskaya

, Sharov

, The frequency dictionary of modern Russian language, Azbukovnik, Moscow, 2009.

27.

Golovin

G.V.

, Receptive vocabulary size measurement for Russian language, Sotcio-Psikhologicheskie Issledovanija3 (2015), 148–159.

28.

Ivanov

V.V.

, Solnyshkina

M.I.

and Solovyev

V.D.

, Efficiency of text readability features in Russian academic texts, In Computational Linguistics and Intellectual Technologies, V. 17, 2018, pp. 277–287.

Prediction of reading difficulty in Russian academic texts

Abstract

Keywords

1 Introduction

1.1 Linear models for Russian academic texts

2 Related works

3 Dataset

3.1 Collecting a corpus of school textbooks

Table 1 Sizes (in tokens) of documents in the RRC corpus Grade BOG NIK Label 5-th – 20,788 5 6-th 17,586 17,392 6 7-th 24,255 25,157 7 8-th 53,077 43,395 8 9-th 45,416 47,212 9 10-th 81,974 42,030 10 10-th* 105,840 – 11 11-th – 41,454 11 11-th* 105,735 – 12

3.3 Sampling from the corpus

Table 2 Number of samples in a collection of size S (number of tokens) Label S=500 S=1000 S=2000 S=4000 5 46 23 11 5 6 77 38 18 9 7 106 54 27 13 8 205 106 53 26 9 196 102 51 25 10 263 137 68 33 11 307 162 81 40 12 227 116 58 29 Total 1427 738 367 180

4.1 Methodology for evaluation of readability prediction

4.2 Results of classification

Footnotes

Acknowledgments

References

Table 1
Sizes (in tokens) of documents in the RRC corpus

Grade BOG NIK Label

5-th – 20,788 5

6-th 17,586 17,392 6

7-th 24,255 25,157 7

8-th 53,077 43,395 8

9-th 45,416 47,212 9

10-th 81,974 42,030 10

10-th* 105,840 – 11

11-th – 41,454 11

11-th* 105,735 – 12

Table 2
Number of samples in a collection of size S (number of tokens)

Label S=500 S=1000 S=2000 S=4000

5 46 23 11 5

6 77 38 18 9

7 106 54 27 13

8 205 106 53 26

9 196 102 51 25

10 263 137 68 33

11 307 162 81 40

12 227 116 58 29

Total 1427 738 367 180