Detecting errors in machine translation using residuals and metrics of automatic evaluation

Abstract

Errors and residuals are closely related measures of the deviation. An error is a deviation of the observed value (PEMT output) from the expected value (MT output), while the residual of the observed value is the difference between the observed and predicted value of quality. We propose an exploratory data technique representing an ideal instrument to evaluate and improve machine translation (MT) systems. The main contribution consists of a rigorous technique (a statistical method), novel to the research of MT evaluation given by residual analysis to identify differences between MT output and post-edited machine translation output regarding human translation (reference). The residual analysis of the automatic metrics can help us to discover significant differences between MT and PEMT and to identify questionable issues regarding the one reference. In this study, we show the usage of residuals in MT evaluation. Using residual analysis, we identified sentences, in which significant differences were found in the scores of automatic metrics between MT output and post-edited (PE) MT output from Slovak into English.

Keywords

Machine translation evaluation residuals analytical language inflectional language MT errors

1 Introduction and related work

Due to the wide-spread development of CAT tools (Computer-Assisted Translation) and MT (Machine Translation) systems and their extensive use in the translation process, the evaluation of MT became more and more important. In MT evaluation, optimisation is closely related to quality. Based on the results of MT evaluation, the examined MT system is optimised. Assessing the quality of MT output can be conducted in two different ways- manually and automatically. The manual or human approach comprises 1) traditional human assessment methods [11 , 29] such as fidelity and intelligibility (used by the Automatic Language Processing Advisory Committee, ALPAC, [16]), or later applied fluency, adequacy and comprehension (Advanced Research Projects Agency, ARPA, [19]). They are often evaluated at the sentence level in a scale from 1 (worst) to 5 (best); 2) advanced methods of human assessment such as ranking [32 –34], it is the official MT evaluation method of WMT (Workshop on Statistical Machine Translation) campaigns since 2007, and similar to fluency/adequacy, is evaluated at the sentence level in scale from 1 (worst) to 5 (best), or task-oriented methods [8 , 18], which are more informative as previous ones, or post-editing [1 , 38] performed by humans (however, there is tendency to automate post-editing) consisting of checking and, when necessary, correcting MT output. Methods of manual MT evaluation require human translator knowledge, assessing the quality of MT output along the two axes- language correctness and semantic fidelity. The human evaluation would be the most desirable, especially the errors analysis, but it is intuitive or subjective, time and labour consuming and also expensive. Vilar et al. [9] pointed out that subjectivity of evaluators causes a problem regarding biased judgments towards MT as well as having no clear definition of a numerical scale by translation quality assessment. To overcome the issues with human evaluation, automatic evaluation metrics have been proposed and became a common way of the MT evaluation because they are fast, cheap, reusable and language-independent. They compute similarity -matches based on comparisons between a set of references (fixed human translations) and the corresponding MT output. Automatic evaluation also requires human interaction, but it is much less time-consuming and expensive compared to human evaluation (texts are translated by humans only once and can be used many times). Automatic metrics can be classified into three groups: based on lexical similarity [3 , 37], they measure the concordance in words within the sequences, word order and also edit distance; based on syntactic similarity [2 , 27], they take sentence structure into account such as part-of-speech, or based on semantic similarity [5 –7], they take synonyms or named entity into account. However, the results (the scores of these metrics) are only quantitative and do not inform or identify the suspicious segments or sentences. The traditional way to evaluate automatic metrics performance is through ranking, i.e. human evaluators are asked to rank sentence by sentence of MT output of different MT systems (usually 4 MT systems, but it depends on language pairs). These rankings using Pearson correlation coefficient or Kendall’s τ are used to evaluate the automatic metrics [32 –34]. They provide only quantitative scores, which MT system provides better translation regarding automatic metrics, but do not give us much detail about the error types which occurred in MT output. Automatic metrics are mainly designed for the evaluation and comparison of MT systems and not for translation quality assessment.

In this study, we present another method, which can be used in MT evaluation – residual analysis. We propose an exploratory data technique representing an ideal instrument to evaluate and improve MT systems. The main contribution consists of a rigorous technique (a statistical method), novel to the research of MT evaluation given by the residual analysis to identify differences between MT output and post-edited machine translation output regarding the human translation (reference).

2 Residuals

Researchers in physical, economic, social or behavioural sciences or medicine even in translation studies (human or machine) are interested in variables which are unobservable, measured with an error. This type of latent variable can be, for example, a quality of machine translation, which is not directly observable. Besides that language is considered as a complex system of features and relationships among them.

Errors and residuals are closely related measures of the deviation. The error is a deviation of the observed value (PEMT/reference) from the expected value (MT), while the residual of the observed value is the difference between the observed and predicted value of the quality: $Residual = Observed - Predicted .$

Residual analysis is one of the ways how to evaluate a model (MT output). It helps us to assess the adequacy of the model [25]. Roger W. Hoerl [36] claimed that analysis of the residuals is an effective method for assessing the fit of the model to the data and determining whether the model is useful. Residuals can be thought of as elements of variation unexplained by the fitted model [31]. For standard (normal) linear models, residuals are used to verify homoscedasticity, the linearity of effects, the presence of outliers, normality and independence of the errors [17]. According to Bollen and Arminger [21], the unstandardized residuals can be plotted to help identify unusual values of the residuals in comparison to the others, while the calculated values of residuals depend on the metric used to measure the observed variables (in our case, metrics of automatic MT evaluation like, metrics of accuracy and error rate).

Evaluation of MT output can be viewed as an evaluation of the validity of assumptions of a statistical model. Residuals allow us to identify patterns, better understand and interpret problems of the model and subsequently eliminate, correct, analyse them or analyse their influence on the quality of machine translation (e.g. mistakes in machine translation). Such analysis has not been widely used or applied for textual data. Topp and Gomez [35] developed residual analysis for censored data (illustrated on a data set from an AIDS clinical trial study). Hildreth [23] showed the use of residual analysis in structural equation modelling (a statistical methodology used to examine causal relationships in observational data), which is used in the social and behavioural sciences due to its ability to model complex systems of human behaviour.

3 Experiment

The goal of this experiment was to identify sentences, in which significant differences were found in the scores of automatic metrics between MT output and post-edited (PE) MT output from Slovak into English. The residual analysis of the automatic metrics can help us to discover significant differences between MT and PEMT and to identify questionable issues regarding the one reference. We calculated automatic scores by relying on one reference. We used our tool to align and compute the automatic scores of metrics of accuracy and error rate. Further, we used these metrics as variables for residual analysis at the sentence level. A MT system - Google Translate (GT) was used. It is a free web translation service (and only one) offering translation from/to a low-resource language (Slovak).

3.1 Dataset

We create a dataset, which covers 360 sentences translated only in one direction- from an inflectional language (Slovak) into an analytical language (English). We chose this direction only for the following reason: when we have translated a text from a foreign language into our mother tongue, we obtained lower scores of automatic metric BLEU (resulted from our pre-research results). If we wanted to achieve a higher score for BLEU metrics, we had to examine vice versa translation (BLEU metric is not suitable for inflective languages, e.g. one English noun has six forms in Slovak differing only in suffixes, and also in comparison to English, the Slovak language has a loose word order). These 360 sentences written in Slovak were translated by a statistical machine translation system and consequently post-edited by two professional translators (P) and 12 master’s students of Translation Studies (S). Reference was created by two translators and one native speaker, whose mother tongue is English but also speaks Slovak. After machine translation and post-editing, our dataset is composed of 360 machine translated sentences and 5040 post-edited sentences.

3.2 Evaluation methods

WER (Word Error Rate, error-based metric) is based on the edit distance, takes into account the word order, and accounts the Levenshtein distance between a hypothesis (MT output) and a reference (human translation). It calculates the minimum number of edits (insertion, deletion, and substitution) needed to transform the hypothesis into the reference, i.e. $WER (h, r) = \frac{{min}_{e \in E (h, r)} (insert (e) + delete (e) + substitute (e))}{| r |},$ where insertion (e) – number of adding words, deletion (e) – number of dropping words, substitution (e) – number of replacements (in sequence or path e), r is a reference of MT output h, and $min_{e \in E (h, r)}$ is a minimal sequence of adding, dropping, and replaced words necessary to transform the MT output (h) into the reference r.

PER (Position-independent Error Rate) is similar to WER, but it does not take into account the word order. It considers the reference and hypothesis as bags of words and counts the number of times that identical words appear in both sentences (MT and reference), i.e. $PER = 1 - \frac{correct - max (0, hypothesis length - reference length)}{reference length} .$

CDER (Cover Disjoint Error Rate) is a measure oriented towards recall but based on the Levenshtein distance. It uses the fact that the number of blocks in a sentence is the same as the number of gaps between them plus one. It requires both hypothesis and reference to be covered completely and disjointly. Only words in the reference must be covered only once, while in the hypothesis they can be covered zero, one or more times, i.e. $CDER (h, r) = \frac{{min}_{e \in E (h, r)} (insertion (e) + deletion (e) + substitution (e) + long jump (e))}{| r |},$ where insertion (e) – number of adding words, deletion (e) – number of dropping words, substitution (e) – number of replacements (in sequence or path e), long jump (e) – number of long jumps, r is reference translation of hypothesis h and $min_{e \in E (h, r)}$ is minimal sequence of adding, dropping and replaced words necessary to transform the MT output (h) into the reference r.

MT can be evaluated by using precision, recall, and f-measure metrics (well known and widely used in natural language processing). They are based on lexical similarity, i.e., correct word matches between MT output (hypothesis) and reference. Precision is the proportion of words in MT that are present in reference, i.e. $precision = \frac{correct MT words}{hypothesis length} .$

Recall is the proportion of words in reference that are present in MT, i.e. $recall = \frac{correct MT words}{reference length} .$

F-measure is a harmonic mean of precision and recall.

BLEU (Bilingual Evaluation Understudy, [20]) is a geometric mean of n-gram precisions (for n-gram of size 1– 4) and the second part is a brevity penalty (BP), i.e. length-based penalty to prevent very short sentences as compensation for inappropriate translation. $\begin{matrix} BLEU (n) = exp \sum_{n = 1}^{N} w_{n} log p_{n} \times BP, \\ p_{n} = {precision}_{n} \\ = \frac{\sum_{S \in C} \sum_{n - gram \in S} {count}_{matched} (n - gram)}{\sum_{S \in C} \sum_{n - gram \in S} count (n - gram)}, \end{matrix}$ and $= {\begin{matrix} 1, & if h > r \\ e^{1 - \frac{r}{h}}, & if h \leq r \end{matrix} .$

S means hypothesis (h) and r reference in the complete corpus C.

The BLEU reflects two aspects of translation quality- adequacy and fluency by calculating word or lexical precision.

We used residuals to compare the scores of automatic metrics of PEMT with MT at the sentence level. In our case the analysis composed of residual analysis is defined as follows: $\begin{matrix} (residual value)_{i} & = & (score of PE sentence)_{i} \\ - (score of MT sentence)_{i}, \end{matrix}$ i = 1, 2, …, I while I is a number of examined sentences in the dataset and PE means post-edited.

We used a rule ±2σ to identify the extreme values: $\begin{matrix} mean of residuals (PE - MT) \pm 2 \\ * st . dev . of residuals (PE - MT), \end{matrix}$ the residual values outside the interval are considered as extreme values.

By aggregating the scores of automatic metrics by weighted average, we created one variable for each metric. The residual values above the residual mean predicate the above average assessment of PEMT using the automatic metric in comparison to MT output. On the contrary, residual values below the residual mean predicate the below average assessment of PEMT output using the automatic metric in comparison to MT output. Identification of extreme values helps us to detect sentences, in which the significant differences between MT and PEMT were found (translation from Slovak into English).

4 Results

Regarding the number of examined sentences (360 sentences for evaluation), we introduce only the first 37 sentences. The graphs (Figs. 1–3) depict the scores of the precision, recall, and f-measure PEMT and MT output from Slovak into English and their residuals^′ values.

Fig.1

Results of residuals analysis of the precision metric of MT and PEMT.

Some sentences translated by MT system are more close to a reference as PEMT ones such as sentences 3, 11 or 14 (green colour). On the other hand, PEMT sentences like sentences 9 or 29 achieved higher scores of the precision than MT sentences.

Sentences 2, 29 and 31 (Figs. 1–3) were above average assessed by the precision, recall, and f-measure metrics regarding the accuracy of PEMT. Besides, sentence 35 (Figs. 2 and 3) based on the recall and f-measure metrics was also above average assessed regarding the accuracy of PEMT. The sentences were the closest to a reference translation. By contrast, sentence 14 (Figs. 1–3) was below average assessed by the precision, recall, and f-measure metrics regarding the accuracy of PEMT against MT. Similarly, sentence 11 (Fig. 1) based on the precision and sentence 18 (Figs. 2 and 3) based on the recall and f-measure were also below average assessed regarding the accuracy of PEMT.

Fig.2

Results of residuals analysis of the recall metric of MT and PEMT.

A significant difference was found in the case of sentence 29 (Figs. 1 and 3) in the value of precision and f-measure between PEMT and MT in favour of PEMT, and in case of sentence 14 (Figs. 2 and 3) in recall and f-measure values in favour of MT.

The graphs (Figs. 4–6) depict the scores of the automatic metrics of error rate (WER, PER, and CDER) of PEMT and MT output from Slovak into English and their residuals' values. The residual values above the residual mean predicate the above average error rate of PEMT using the automatic metric in comparison to MT output. On the contrary, residual values below the residual mean predicate the below average error rate of PEMT output using the automatic metric in comparison to MT output. Identification of extreme values helps us to detect sentences, in which significant differences in the error rate between MT and PEMT were found (translation from Slovak into English).

Fig.3

Results of residuals analysis of the f-measure metric of MT and PEMT.

Fig.4

Results of residuals analysis of the PER metric of MT and PEMT.

Sentence 14 (Figs. 4–6) had an above average error rate determined by the PER, WER, and CDER metrics. Moreover, sentence 18 (Fig. 4), based on the PER and sentence 35 (Fig. 5), also based on the WER metric, also had an above average error rate. On the other hand, the PEMT of sentences 2 and 29 (Figs. 4–6) had a below average error rate determined by the PER, WER, and CDER metrics. Likewise, based on the PER, PEMT of sentences 31 and 35 (Fig. 4) had a below average error rate.

Fig.5

Results of residuals analysis of the WER metric of MT and PEMT.

Fig.6

Results of residuals analysis of the CDER metric of MT and PEMT.

Fig.7

Values of the BLEU-1 metric of PEMT and MT English output.

Fig.8

Results of residuals analysis of the BLEU-1 metric of MT and PEMT.

Fig.9

Values of the BLEU-2 metric of PEMT and MT English output.

A significant difference was found in the case of sentence 14 (Figs. 4 and 6) in the PER and CDER, and in the case of sentence 35 (Fig. 5) in the value of the WER metric between the PEMT and MT in favour of MT output. On the other hand, a significant difference was found in the values of WER and CDER for sentence 29 (Figs. 5 and 6), and for sentence 2 (Fig. 5) in WER between PEMT and MT in favour of PEMT output.

The graphs visualise the values of the automatic metric of accuracy, i.e. BLEU-n, (n = 1 to 4) of the PEMT and MT output (Figs. 7, 9, 11 and 13). The graphs (Figs. 8, 10, 12 and 14) depict the residuals’ values of BLEU metric. Similar to precision, recall and f-measure, the residual values above the residual mean predicate the above average accuracy of PEMT using the automatic metric in comparison to MT output. On the contrary, residual values below the residual mean predicate the below average accuracy of PEMT output using the automatic metric in comparison to MT output. Identification of extreme values helps us to detect sentences, in which the significant differences in accuracy between MT and PEMT were found.

Fig.10

Results of residuals analysis of the BLEU-2 metric of MT and PEMT.

Fig.11

Values of the BLEU-3 metric of PEMT and MT English output.

Fig.12

Results of residuals analysis of the BLEU-3 metric of MT and PEMT.

Fig.13

Values of the BLEU-4 metric of PEMT and MT English output.

The PEMT of sentence 29 (Figs. 8, 10, 12 and 14) was above average assessed by BLEU-n (n = 1 to 4). Besides, based on the score of BLEU-1 the PEMT of sentences 31 and 35 (Fig. 8) were above average assessed regarding accuracy. Further, based on the BLEU-2, the PEMT of sentence 10 was above average assessed regarding accuracy (Fig. 10). Similarly, the PEMT of sentence 25 (Figs. 12 and 14) was above average assessed by the BLEU-3 and BLEU-4 metric. On the contrary, based on the scores of the BLEU-n (n = 1 to 4), the PEMT of sentences 14 and 18 were below average assessed in terms of accuracy (Figs. 8, 10, 12 and 14). These sentences were more similar to reference when they were translated by machine than post-edited. Likewise, the PEMT of sentence 19 (Fig. 14) in respect of the score of the BLEU-4.

Fig.14

Results of residuals analysis of the BLEU-4 metric of MT and PEMT.

The significant difference was found in sentence 29 in the scores of the BLEU-n (n = 1 to 4) between PEMT and MT in favour of PEMT output (Figs. 8, 10, 12 and 14). On the other hand, a significant difference was found in sentence 14 (Figs. 8, 10, 12 and 14) in the scores of the BLEU-n (n = 1 to 4) between MT and PEMT in favour of MT output. In other words, MT was more similar to reference than PEMT.

5 Conclusion

The aim of the study was an examination of two variables – post-edited machine translation (PEMT) output and machine translation (MT) output, both through the residuals and the use of reference. Based on the residual analysis of automatic evaluation (using metrics of automatic MT evaluation) of PEMT and MT, we found significant differences in the score of automatic MT metrics of PEMT and MT at the sentence level (Figs. 1–6, 8, 10, 12 and 14). A significant difference was found in the sentence 29 in favour of PEMT and vice versa in sentence 14 in favour of MT (within the recall and precision of the translation).

Sentence 29:

ST: Strojový preklad sa často používa ako:

MT: Machine translation is often used as:

Ref.: Machine translation is often used as:

PEMT_S1: Machine translation is often used as:

PEMT_P1: Machine translation is often used as:

Sentence 14:

ST: Ročné náklady EÚ na preklady dokumentov pre jednotlivé krajiny v roku 2007 predstavovali približne 400 miliónov eur.

MT: The annual cost to the EU of translations of documents for each country in 2007 amounted to approximately € 400 million.

Ref.: The EU’s annual document translation cost for each country in 2007 amounted to approximately € 400 million.

PEMT_S2: The annual cost to the EU for translations of documents for each country in 2007 amounted to approximately € 400 million.

PEMT_S9: The annual cost of translations of documents for the EU for each country in 2007 amounted to approximately 400 million €.

PEMT_P1: The annual EU cost of the translated documents for each country was approximately € 400 million in 2007.

Due to the accuracy and coverage of the words, a significant difference was in sentence 29 in favour of the PEMT and sentence 14 in favour of the MT. Sentence 29 is very specific, it was correctly translated by MT system, but the sentence was too short (consisting of six words), which resulted in a penalty for shortness. MT sentence 14 is covered by more words, than the reference 14. Additionally, more words in MT, regardless of the syntax, are the same as in the PEMT sentence than in the reference.

Due to the error rate, a significant difference was in sentences 29 and 2 in favour of PEMT.

Sentence 2:

ST: Strojový preklad (Machine Translation, d’alej len MT) sa stal v poslednom desat’ročí významným predmetom výskumu nielen v akademickej sfére, ale aj v komerčnej sfére.

MT: Machine translation (Machine Translation, hereinafter MT) has become in the last decade significant research subject not only in academia, but also in the commercial sphere.

Ref.: Over the last decade, machine translation (MT) has become an important area of study and research, for both commercial and academic purposes.

PEMT_S12: In the last decade machine translation (Machine Translation, hereinafter MT) has become a significant subject of research not only in academia, but also in the commercial area.

PEMT_S10: In the last decade machine translation (hereinafter MT) has become a significant subject of research not only in academic sphere, but also in the commercial sphere.

PEMT_P1: In the last decade machine translation (Machine Translation, hereinafter MT) has become significant research subject not only in academic, but also in the commercial sphere.

In the case of the MT of sentence 2, there is a strong sentence structure and lexicon (vocabulary) is known. It did not cause a problem through the minimal edit operations to transform the MT sentence to correct one.

Based on the residuals we can only analyse the sentence with extremes and then identify the major MT errors. With a closer look at the reported extreme sentences in the whole dataset we could identify several relevant MT errors. In our case, the most frequent MT errors composed of syntactical (word order and agent), lexical (omitted, mistranslated and synonyms), morphological errors (articles and passive), and punctuation.

Residual analysis is very useful if we do not want to make error analysis of the whole MT output which is not only time-consuming and expensive but often disputable. Our approach to the evaluation of MT systems (using automatic MT metrics) is original, representing an ideal instrument to evaluate and improve MT systems. Besides, this extreme identification (using residuals) can help us to detect major MT errors.

Footnotes

Acknowledgments

This work was supported by the Slovak Research and Development Agency under the contract No. APVV-14-0336.

References

Guerberof Arenas

, Correlations between productivity and quality when post-editing in a professional context, Machine Translation28(3-4) (2014), 165–186.

Han

A.L.-F.

, Wong

D.F.

and Chao

L.S.

, A robust evaluation metric for machine translation with augmented factors, Proceedings of COLING (2012), 441–450.

Chen

, Kuhn

and Foster

, Improving amber, an mt evaluation metric, Proceedings of the 7th Workshop on Statistical Machine Translation2012, pp. 59–63.

Tillmann

, Vogel

, Ney

, Zubiaga

and Sawaf

, Accelerated dp based search for statistical translation, Proceeding of Eurospeech (1997), 2667–2670.

C.-K.

and Wu

, Meant: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility via antic frames, Proceedings of ACL (2011), 220–229.

C.-K.

and Wu

, Structured vs. flat semantic role representations for machine translation evaluation, Proceedings of the 5th Workshop on Syntax and Structure in Statistical Translation (2011), pp. 10–20.

C.-K.

, Turmuluru

A.K.

and Wu

, Fully automatic semantic mt evaluation, Proceedings of the 7th Workshop on Statistical Machine Translation2012, pp. 243–252.

Voss

C.R.

and Tate

R.R.

, Task-based evaluation of machine translation (mt) engines: Measuring how well people extract who, when, where-type elements in mt output, Proceedings of 11th Annual Conference of the European Association for Machine Translation2006, pp. 203–212.

Vilar

, Xu

, D’Haro

L.F.

and Ney

, Error analysis of statistical machine translation output, Proceedings of the 5th International Conference on Language Resources and Evaluation2006, 697–702.

10.

Doddington

, Automatic evaluation of machine translation quality using n-gram co-occurrence statistics, HLT Proceedings (2002), 138–145.

11.

Toury

, Descriptive Translation Studies and Beyond, 1995. Amsterdam and Philadelphia: John Benjamins Publishing Company.

12.

Toury

, In Search of a Theory of Translation, Tel Aviv: The Poter Institute for Poetics and Semiotics, 1980.

13.

Giménez

and Márquez

, Linguistic features for automatic evaluation of heterogenous mt systems, Proceedings of the Second ACL Workshop on Statistical Machine Translation2007, 256–264.

14.

House

, Translation quality assessment: Past and present, London and NY: Routledge, 2015.

15.

Laoudi

, Tate

R.R.

and Voss

C.R.

, Task-based mt evaluation: From who/when/where extraction to event understanding, Proceedings of LREC06 (2006), 2048–2053.

16.

Carroll

J.B.

, An experiment in evaluating the quality of translation, Mechanical Translation and Computational Linguistics9(34) (1966), 67–75.

17.

Nobre

J.S.

and da

, Motta Singer, Residual analysis for linear mixed models, Biometrical Journal49(6) (2007), 863–875.

18.

White

J.S.

and Taylor

K.B.

, A task-oriented evaluation metric for machine translation, pp, First International Conference on Language Resources & Evaluation (1998), 21–25.

19.

Church

and Hovy

, Good applications for crummy machine translation, Machine Translation8(4) (1993), 239–258.

20.

Papineni

, Roukos

, Ward

and Zhu

W.-J.

, Bleu: A method for automatic evaluation of machine translation, pp, Proceedings of ACL (2002), pp. 311–318.

21.

Bollen

K.A.

and Arminger

, Observational residuals in factor analysis and structural equation models, Sociological Methodology21 (1991), 235–262.

22.

K.-Y.

, Wu

M.-W.

and Chang

J.-S.

, A new quantitative quality measure for machine translation systems, Proceedings of the 14th Conference on Computational Linguistics, 1992, 433–439.

23.

Hildreth

L.A.

, Residual analysis for structural equation modelling, Graduate Theses and Dissertations 13400, http://lib.dr.iastate.edu/etd/13400.

24.

Koponen

and Salmi

, On the correctness of machine translation: A machine translation post-editing task, Journal of Specialised Translation23 (2015), 118–136.

25.

Munk

, Drlik

and Vrabelova

, Probability modelling of accesses to the course activities in the web-based educational system, Computational Science and its Applications – ICCSA 20116768 (2011), 485–499.

26.

Plitt

and Masselot

, A productivity test of statistical machine translation post-editing in a typical localisation context, The Prague Bulletin of Mathematical Linguistics93 (2010), 7–16.

27.

Popovic

and Ney

, Word error rates: Decomposition over pos classes and applications for error analysis, Proceedings of the Second ACL Workshop on Statistical Machine Translation (2007), pp. 48–55.

28.

Snover

, Dorr

, Schwartz

, Micciulla

and Makhoul

, A study of translation edit rate with targeted human annotation, Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (2006), pp. 223–231.

29.

Williams

, Translation quality assessment*, Mutatis Mutandis2(1) (2009), 3–23.

30.

Aranberri

, Labaka

, Díaz de Ilarraza

, and Sarasola

, Comparison of post-editing productivity between professional translators and lay users, Proceedings of the Third Workshop on Post-Editing Technology and Practice2014, pp. 20–33.

31.

NIST/SEMATECH, e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/.

32.

Bojar

, Buck

, Callison-Burch

, Federmann

, Haddow

, Koehn

, Monz

, Post

, Soricut

and Specia

, Findings of the workshop on statistical machine translation, Proceedings of the eighth Workshop on Statistical Machine Translation2013, pp. 1–44.

33.

Bojar

, Buck

, Federmann

, Haddow

, Koehn

, Leveling

, Monz

, Pecina

, Post

, Saint-Amand

, Soricut

, Specia

and Tamchyna

, Findings of the workshop on statistical machine translation, Proceedings of the Ninth Workshop on Statistical Machine Translation (2014), pp. 12–58.

34.

Bojar

, Chatterjee

, Federmann

, Haddow

, Huck

, Hokamp

, Koehn

, Logacheva

, Monz

, Negri

, Post

, Scarton

, Specia

and Turchi

, Findings of the workshop on statistical machine translation, Proceedings of the Tenth Workshop on Statistical Machine Translation (2015), pp. 1–46.

35.

Topp

and Gómez

, Residual analysis in linear regression models with an interval-censored covariate, Statistics in Medicine23(21) (2004), 3377–3391.

36.

Hoerl

R.W.

, The Reality of Residual Analysis, Quality Progress, 2008.

37.

Banerjee

and Lavie

, Meteor: An automatic metric for mt evaluationwith improvedcorrelationwith human judgments, Proceedings of the ACL (2005), 65–72.

38.

O’Brien

, Balling

L.W.

, Carl

, Simard

and Specia

eds., Post-editing of Machine Translation: Processes and Applications, Newcastle upon Tyne: Cambridge Scholars Publishing, 2014.