Using Artificial Intelligence to Predict Decisions of the Turkish Constitutional Court

Introduction

The quest for the holy grail of reducing legal uncertainty has gained a new momentum with recent advances in artificial intelligence (AI)-based techniques that have brought about auspicious outcomes when applied to the esoteric domain of judicial decision making. Among these techniques, machine learning (ML) and deep learning (DL) have attracted the attention of researchers as they provide us with devices to predict judicial outcomes derived from the semantic of law and case texts (Surden, 2014). ML tools are not suitable to make causal inferences but they can be effectively utilized to predict the most likely outcome of a given observation (Dumas & Frankenreiter, 2020, pp. 62–63). Setting aside the normative question of whether or not the prediction of judicial outcomes is desirable, especially from the rule of law and procedural justice perspectives, computational law or computational legal analysis, as the field has come to be known, may boost legal certainty and conduct of judicial proceedings in a timely fashion (Frankenreiter & Livermore, 2020; Livermore, 2019).

This article presents the first systematic study on predicting the outcome of cases adjudicated by the Turkish Constitutional Court (TCC) through mining the available textual information. We apply text classification using DL models as the AI technique to predict the case outcomes of individual applications to the TCC regarding public morality and freedom of expression issues. The research omits constitutionality review litigation because it often includes multiple and diverse issues in each case and its texts were not well-structured until recently.

We have chosen limiting our analysis to public morality and freedom of expression cases as they usually involve highly controversial features, which make them particularly attractive subjects for the legal community striving for decreasing legal uncertainty. In the research, public morality involves obscenity, public indecency, dismissal from public service on grounds of “immoral conduct,” sex work, sexual orientation, and gender identity cases. The relationship between public morality and law raises pivotal challenges for courts when they deal with certain forms of conduct of consenting adults. Public morality incorporates laws that restrict these conducts on the grounds that they are morally wrong (Wolfe, 2000). Many constitutions and core international and regional human rights conventions recognize public morality as a legitimate basis for limiting the exercise of certain rights (Thoreson, 2018). Regarding freedom of expression cases examined in the research, they encompass all complaints about interferences with freedom of speech, freedom of the press, and access to the Internet. As is well-known, freedom of expression is one of the main linchpins of a democratic constitutional order and governmental interferences with the use of this right usually represent a challenge to democratic rule and human rights.

This article hopes to contribute to computational law by drawing on a DL method as an AI technique for text classification to predict outcomes of court cases. Our research goal is to construct a predictive model for individual applications to the TCC regarding public morality and freedom of expression. The research is premised on the idea that information extracted directly from published decisions can be used for this purpose with high accuracy. The outcome prediction problem is framed as a binary classification task where the court delivers a violation or a nonviolation decision. We include “inadmissible” rulings as part of nonviolation decisions, which involve a procedural examination prior to undertaking review on the merits. While ML techniques like support vector machine (SVM) are widely used in predicting courts’ rulings, we prefer applying DL to predict case outcomes as we seek to demonstrate that DL method like multilayer perceptron (MLP) with the appropriate model parameters can successfully work even with a relatively small data set.

It is claimed that many AI models cannot yield the same success rate found in English-based corpus because of agglutinative features of Turkish (İstanbul Barosu, 2019). In agglutinative languages like Turkish, word forms consist of morphemes concatenated to a root morpheme or to other morphemes, which creates challenges for applications of natural language processing (NLP) techniques (Oflazer, 2018). However, using sentiment analysis, an NLP technique, researchers achieved 75% accuracy on binary classification of movie reviews in Turkish (Gezici & Yanıkoglu, 2018). Within this context, the research aims to illustrate that DL techniques can also be utilized in making relatively accurate predictions on legal texts in Turkish.

This article first reviews the relevant literature. The following section gives a sketch of the structure and powers of the TCC. Next, we present the methodology of the research. Then, we apply MLP to our data set and report the results. This article concludes with a discussion of the findings.

Related Works

Predicting judicial outcomes have their pedigree in the research that seeks to account for judicial behavior. Researchers mainly used case characteristics and votes of justices to develop models of judicial decision making but their work tended to focus on cases already decided rather than forecasting future outcomes (Martin et al., 2004). In one of the early studies, relying on a statistical model, researchers were able to predict 75% of the U.S. Supreme Court’s cases correctly during the court’s 2002 term better than the legal experts who forecast 59.1% of the cases accurately (Ruger et al., 2004). Another early study using ML reported a 91.8 accuracy in automatically classifying and predicting cases (Ashley & Brüninghaus, 2009). Recently, Katz et al. (2017) achieved a success rate of 70% in predicting case rulings of the U.S. Supreme Court on the basis of information extracted from legal texts.

Research on predicting judicial outcomes has also extended into analyzing other national courts’ decisions, which has provided an opportunity to utilize ML and DL in different languages other than English. For example, using ML techniques to predict the ruling of the French Court of Cassation and the law area to which a case belongs to, Şulea et al. (2017) reported results of 98% average F1 score in predicting a case ruling and 96% F1 score for predicting the law area of a case. Likewise, Bayesian networks were successfully used to classify decisions of a Brazilian Labor Court to forecast the success rate of employers and employees in their litigation (Barros et al., 2018). Another Brazilian investigation using a data set with 4,043 cases was able to predict 79% of the cases correctly (Lage-Freitas et al., 2019). An Indian study revealed 78% accuracy in predicting the verdict of selected judicial cases (Bhilera et al., 2019). Indian researchers also reported successful results for predicting Delhi District Court cases (Shaikh et al., 2020). Comparable successful predictions were also reported by Chinese researchers (Liu & Chen, 2017).

In addition to national courts, researchers have explored cases of supranational courts applying ML. For example, Aletras et al. (2016) employed SVM to predict whether cases violate Articles 3, 6, and 8 of the European Convention on Human Rights. They extracted the textual information from relevant sections of the European Court of Human Rights (ECtHR) decisions and managed accuracy over 79% on average performed on 584 cases separated by subjects. In another study on the ECtHR, investigators applied SVM to predict its rulings. They excluded the parts of the decisions that contain the arguments and achieved an average accuracy of 75% in predicting the violation of nine articles of the European Convention (Medvedeva et al., 2019). O’Sullivan and Beel (2019) also used ML models to predict whether an application has led to a violation or a nonviolation of the decision by the ECtHR. They achieved an accuracy of about 69% across the 12 Articles of the European Convention.

On the other hand, some studies have disclosed relatively lower levels of accuracy in predicting judicial outcomes applying ML. For example, an experiment was carried out to forecast the Philippine Supreme Court case decisions yielding an accuracy of 45% on n-grams data sets and 55% on topic data sets (Benedict et al., 2018). The researchers attributed this poor performance mainly to the lack of a standard format on the penning of decisions by the Philippine Supreme Court (Benedict et al., 2018).

There have also been studies to evaluate the performance of various machine classifiers in predicting ideological direction, that is, “conservative decision” or “liberal decision” of U.S. Circuit Courts (Hausladen et al., 2020). A similar study explored a range of text-scaling methods for the analysis of jurisprudential change in the German Federal Constitutional Court opinions on European integration (Dyevre, 2020).

Compared to widely used ML techniques, applications of DL methods in judicial settings seem to have very recently come to the attention of researchers. For instance, Undavia et al. (2018) reported 72.4% accuracy when classifying the U.S. Supreme Court decisions into 15-broad Supreme Court Database categories. In addition, Chalkidis and Kampas (2019) presented a study focusing comprehensively on text classification, information extraction, and information retrieval. However, these studies did not involve the prediction of judicial decisions.

TCC

Founded in 1962, the TCC plays a crucial role in political, economic, and social life, the rulings of which have attracted both praise and heavy criticisms from different segments of the society (Oder, 2019). It is composed of 15 justices mostly appointed by the president with a fixed term of 12 years and structurally operates in the form of a plenary, sections, and commissions. Article 146 of the Constitution holds that the president shall directly appoint 12 justices of the TCC, while the parliament elects the remaining three justices. The court is vested with significant powers, which makes it a crucial actor in political life (Özbudun, 2011). Article 148 of the Turkish constitution stipulates the functions and powers of the court. On this basis, it examines the constitutionality of laws in abstract and concrete norm reviews. It has also the competences of dissolving political parties and acting as a Supreme Criminal Tribunal for trying certain high-ranking public officials. The court’s function as a judicial review body has acquired a new significance following the introduction of a presidential system in 2018, as it has the power to oversee presidential decrees. Finally, the court has also the power of deciding individual applications (a form of constitutional complaint). Under this remedy, a person may apply to the Constitutional Court alleging that the public power has violated any one of their fundamental rights and freedoms secured under the constitution which falls into the scope of the European Convention on Human Rights. As we have seen, the court has a wide range of competencies but it is the individual application remedy that constitutes the bulk of the court’s workload. ¹ Since the individual application system provides people with reasonably easy access to the court, it has led to a considerable rise in the court’s docket. There have been about 285,000 individual applications since 2012 when the TCC began to admit them. Although the court has concluded about 86% of all the individual complaints, this has not prevented a backlog of applications in the court proceedings mostly because it received around, on average, 40,000 individual applications per year. ²

The individual application system operates based on the principle of subsidiarity, entailing that an application may only be lodged following the exhaustion of all other legal remedies. ³ The assessment of individual complaints is subject to a two-tiered process: admissibility and review on the merits. ⁴ Admissibility decisions are, as a rule, rendered by the two-judge composed commissions. The aim of the admissibility procedure is to examine whether the application falls within the jurisdiction of the TCC and merits further examination. If an application is found inadmissible, the court does not proceed to examining it on the merits and the case is closed. ⁵ By contrast, if application is deemed to be admissible, it is heard by the two sections of the court consisting of seven members and, in some cases, by the full court on the merits rendering either a violation or a nonviolation judgment. The sections are entitled to bring the case before the plenary if the application in question has the potential of creating a precedent and a landmark ruling. Both the sections and the full court can also undertake admissibility review in tandem with a review on the merits. The applicant cannot appeal the decision of the TCC as it is final, but they can take the case before the ECtHR.

If the TCC finds a violation of a constitutional right, it may only declare the unconstitutionality of the judicial decision or acts of public bodies as it does not have the power of annulling the law and declaring it null and void in the individual application cases. ⁶ Even if the court finds a violation of the constitution, it is not allowed to overrule the decision of the ordinary court. It may refer the case back to the ordinary court for a retrial or may award nonpecuniary damages for the infringement of the constitutional rights of the applicant. The court has declared a violation of a constitutional right judgment in approximately 4% of all the applications but if an application is declared admissible, the success rate amounts to 93% on the merits review conducted by the sections and the general plenary of the court. ⁷

Data

Composing Data Sets

Two data sets with 92 public morality and 388 freedom of speech decisions issued by the TCC between 2012 and 2020 have been created with their subject and reasoning sections. The data sets are separated into two sub–data sets, each of which includes a different section of the case and decision section as seen in Table 1.

OPEN IN VIEWER

Table 1.

Data Sets of the Study.

Data Set	Rulings	Included Sections
Data Set 1a	Public morality	Subject + decision
Data Set 1b	Public morality	Reasoning + decision
Data Set 2a	Freedom of expression	Subject + decision
Data Set 2b	Freedom of expression	Reasoning + decision

All the court’s public morality and freedom of expression rulings in individual applications have been converted to binary as 1 if there is a violation decision, 0 if there is no violation or the application is found inadmissible. We have integrated inadmissible decisions into “noviolation” judgments. If we use three groups instead of two in the research, the ML model cannot make sufficiently accurate predictions since the number of cases that result in nonviolation and “inadmissible” decisions are quite low. Therefore, these cases have been taken as a group by coding 0. In other words, the model predicts as either violation or “nonviolation or inadmissible.” In this way, the model has been established using all available information without any data loss.

Since, we have a relatively limited number of cases confined to public morality issues, it is possible to face with unbalanced data set problem which can negatively affect the learning process. In order to generate balanced data set, the number of cases has been equally distributed into the data set. Therefore, the places of the cases resulted as violation have been changed in the data set. For example, the data set has been arranged like “one of every five cases resulted as violation.” As a result, training and test data sets have an equal proportion of cases concluded as violation.

Research Methodology

We have made predictions that belong to data sets (1a, 1b, 2a, and 2b) by using MLP method, which is one of the DL methods. However, the collected and composing data sets need a series of procedures in NLP method before using any classification method on the data sets. On each data set, preclassification and fundamental word embeddings steps have been made, and the document has been normalized. Bag of Words (BoW) and Term Frequency—Inverse Document Frequency (TF-IDF) scorings have been performed on the normalized document, then vectors obtained have been randomly divided into two as training and test for supervised learning technique. The MLP model has been developed by using training and validation data set and applied to test data set. Consequently, the MLP model performance and results have been achieved for each data set. The stages and methods used in the study are provided in Figure 1. This study uses all cases related to general morality and freedom of speech discussed in the Constitutional Court. However, generating balanced data set is a problematic issue due to the number of cases is restricted. For handling this problem, stratified k-fold cross-validation was used. The stages and methods that used in the study are briefly explained with subheadings.

OPEN IN VIEWER

Figure 1.

Research methodology.

Word Embeddings Steps

Word embeddings, which convert text into numerical vectors, constitute a significant part of text classification methods. This enables us to make linear algebraic operations with texts. In this study, the basic word embeddings steps are used in order to prove even basic word embedding methods can be used in an AI method successfully if the texts are well structured like law text. Therefore, two frequency-based embedding methods, BoW and TF-IDF methods, have been used and compared for this purpose.

BoW model

This model is the most basic form of converting a text into numbers. In this model, term position and word order are not considered. A word takes a numerical value based on how many times it passes in a document. Thus, the document-term matrix consists of words and their number of appearances in the document.

TF-IDF scoring

This method uses weighting to convert text into vectors. Weight is a statistical measure, and it is used to evaluate the importance of a word in a corpus. TF-IDF weight consists of two terms. TF is obtained by dividing the number of times a word appears in a document by the total number of words in that document. The IDF is the logarithm of the total number of documents divided by the number of documents containing the word.

t f i , j = n i , j ∑ k n i j ,

1

w i , j = t f i , j × log N df i ,

2

AI

AI, which is a very detailed and broad concept, is an umbrella notion that can be explained as the ability of machines to think like humans, learn from data, and infer and react like a person with knowledge in that field (Wang & Siau, 2019). Undoubtedly, there are too many methods or subheadings in such a comprehensive and developing concept. Therefore, explaining basic methods and subheadings of the AI is the best way to understand AI concept. ML is one of this subtitles of AI. Accordingly, machines can make inferences from data with the help of various mathematical- and statistical-based algorithms. Within the framework of these developed algorithms, machines can automatically perform various tasks or experiments. One of the most important points in ML, which includes many different methods, is the process of learning data. In order to solve a problem in ML, it is necessary to create the best feature sets that will represent the problem. This consists of the extraction of the features and the selection of the most representative of these features. So feature engineering is very important. Otherwise, the algorithms will not be able to learn at the desired level and will not be able to make a proper inference (Zheng & Casari, 2018).

DL is a subbranch of ML. One of the most important points that distinguish DL from ML is that there is no need for feature sets in DL. In DL, by taking the data as they are, the important data can be automatically determined from the unimportant (or less important) ones (Chollet & Allarie, 2018). Thus, it becomes more possible for machines to make inferences by using a large number of and/or complex data. In this way, AI methods can be used with different types of data such as sound and images. DL allows computers to construct more complex concepts than simple ones (Goodfellow & Courville, 2016). Techniques such as image processing and NLP are developed with DL and applied in many fields such as medicine, finance, and technology. In summary, as can be seen in Figure 2, AI, a concept that started in the 1950s, includes ML introduced in the 1980s and DL that started to be seen in the 2010s with the development of computer science.

OPEN IN VIEWER

Figure 2.

Relationship between the artificial intelligence, machine learning, and deep learning.

idf w = log N df t .

3

Artificial neural networks (ANN) are at the origin of all DL methods. Therefore, the most important point in DL is to design the multilayer artificial network structure, namely the model architecture (number of layers and its contents) and its parameters (hyperparameters) in the best and efficient way. DL includes many different methods. However, as the MLP is the quintessential DL method, we used it in the research (Goodfellow & Courville, 2016).

MLP

MLP is one of the most traditional types of DL techniques, based on ANN, inspired by the way biological nerves work in the human brain (Shanmuganathan, 2016). In general, working principle of the artificial neuron called perceptron (a mathematical model of a biological neuron) is as follows: It collects the signals received from the previous neurons and transmits a signal to another neuron as soon as the total signal accumulation exceeds a certain threshold value (Krenker et al., 2011). Therefore, an artificial neuron or perceptron has fundamental elements that are inputs, weights and bias, weighted sum, activation function, and output. The value obtained for the output of a neuron with these elements is given in Equation 4.

y j = f ∑ i = 1 n w j i x i + b .

4

Inputs (x_i) are information from other neurons. In other words, it is the output of the neurons before it. Weights (w_ji: the connecting weight between neurons of layers j and i) show the importance of information coming to a neuron and its effect on a neuron. Inputs from other neurons are multiplied by the weight of the connections they come from and transmitted to the nucleus. In this way, the effect of the inputs on the output to be produced can be adjusted. Here, weighted sum is obtained by multiplying and summing all inputs to the neuron with their weights and adding the bias value (b) in addition. This value is inserted into a specified activation function (f) to obtain the output value (y_i). The activation function determines the neuron’s output corresponding to this input by processing the weighted sum value in the neuron.

The example of the network structure of an MLP model and the operation of a perceptron that takes place in the MLP model have been presented in Figure 3. As can be understood from Figure 3, an MLP model consists of many perceptrons. MLP is a type of feed-forward neural networks and consists of three main layers (Yang et al., 2016). These layers are input layers that transmit external inputs to hidden layers, hidden layers with many layers and neurons, and the output layer where results are obtained. The features of the data to be learned at the input layer are given as input. Therefore, the data have as many input neurons as the number of features. The hidden layers are located between the input and output layers. How many layers it will consist of and how many neurons there will be in each are variable and are properly determined by the model builder, taking variables such as the problem’s complexity and computation time into account. In the output layer, the class or values of the data to be learned from the model are calculated as output. Nodes apart from input layers use a nonlinear activation function.

OPEN IN VIEWER

Figure 3.

An example of multilayer perceptron model (top) and the operation of a perceptron (bottom).

MLP utilizes a supervised learning technique and its algorithm applies the generalized delta rule consisting of feed-forward and back propagation stages. Accordingly, depending on the value of the error obtained in each solution, the parameters, weights, and biases in the network are updated repeatedly and the closest value to the real value is obtained. The accuracy of the developed network is calculated using the cost or loss function. The aim is to minimize the cost function. Gradient descent algorithm is generally used for this purpose. Updating the weights using the gradient of the cost function is expressed as backpropagation.

One of the important subjects in any learning method like MLP is hyperparameters (Bergsta et al., 2011). Parameters that vary according to the problem and data set are called hyperparameters. How and what value it should take is determined by the designer of the model. Major hyperparameters that arranged in the study can be explained as minibatch size, learning rate, optimization algorithm, epoch number, initial weight values, activation function, number of neurons and layers in hidden layers, and dropout.

Experiments

In the experiment phase, as in every NLP study, preclassification steps and word embeddings steps have been applied to the data sets, which have been made convenient for MLP (see the fourth section). To this end, the CountVectorizer and TfidfVectorizer classes in scikit-learn (library in Python) have been employed to convert the normalized document to the BoW matrix and TF-IDF feature matrix, and the document-term matrixes of them have been created.

In this study, four different data sets have been used as explained in the third section. It is clear that a unique MLP model should be designed for each data set because each data set has different features. The model should be customized by using different hyperparameters according to data set features as explained in the fourth section. Therefore, four different MLP models have been generated.

As this study also aims to compare the performances of BoW and TF-IDF methods, which are the basic word embeddings methods in a Turkish NLP study in which MLP method is used in the classification stage. For this purpose, two separate document-term matrixes were created from each data set by applying the BoW method to one and TF-IDF method to the other. But, since these methods basically use the same original data sets, the same MLP model has been employed for the two sub–data sets created. As a result, we have prepared four models for a total of eight sub–data sets.

We have arranged the document-term matrixes for the supervised learning techniques and we have randomly divided the data sets into 80% training and 20% testing. The models have been created through the train data set and used on the test data set. It was thus determined how accurately the models worked. In addition, stratified k-fold cross-validation, a method used to evaluate the success of learning models, was used in the study. In the stratified k-fold cross-validation method, the train data set is divided into k equal parts. The performance of the model is observed by running the model with one of these parts as a validation data set and the others as a train data set. Then, the process continues in an iterative way, with another part being the validation data set and the others as the train data set. In this way, every data in the train data set are included in both train and validation (Figure 4). This is particularly advantageous for small data sets. Using these data sets, the performance of the model on the test data set is accurately obtained.

OPEN IN VIEWER

Figure 4.

k-Fold cross-validation process (https://scikit-learn.org, 2020).

In this study, k is determined as 5. According to this, any train data set is divided into five different subsets, four of which are train and one of which is validation data. Thus, the average accuracy value resulting from five experiments was found. Thanks to this, it becomes possible in the separation of train and test data to see if the successful performance of the model was coincidental and at the same time minimize problems such as overfitting and underfitting. We use keras which is a python library written for DL. The developed models by using each data set are explained below with sub-titles.

Model for Data Set 1a

The developed model, which is named Model 1a, for the Data Set 1a. This includes subject and decision (as binary) parts of public morality–related cases and comprises four hidden layers apart from input and output layers. Dropouts with a value of 0.5 have been applied to the second hidden layer and 0.3 to the third hidden layer. The input layer nodes number of the model consists of the number of words in the document-term matrix, which is 563 for this model. The batch size has been taken very small, because of the number of data. The architecture and hyperparameters values of the developed Model 1a are presented in Table 2.

OPEN IN VIEWER

Table 2.

Model Architecture of the Model 1a.

	Number of Nodes	Activation Function
Input layer	563
Hidden Layer 1	10	Sigmoid
Hidden Layer 2	30	Sigmoid
Dropout (0.5)
Hidden Layer 3	120	Relu
Dropout (0.3)
Hidden Layer 4	60	Relu
Output layer	1	Sigmoid
Learning rate: 0.0005
Loss function: Binary crossentropy
Optimizer: Adam
Metrics: Accuracy
Epoch number: 200
Batch size: 2

Model for Data Set 1b

The Data Set 1b includes reasoning and decision (as binary) parts of public morality–related cases. The Model 1b that has been developed for this data set contains four hidden layers apart from input and output layers. Also, dropouts with a value of 0.5 have been applied to the second and fourth hidden layers. The input layer nodes number of the model is 9,838. Because this part of cases is quite broad and has too many words, therefore, the size of the document-term matrix is very large. The architecture and hyperparameters values of the developed Model 1b are demonstrated in Table 3.

OPEN IN VIEWER

Table 3.

Model Architecture of the Model 1b.

	Number of Nodes	Activation Function
Input layer	9,838
Hidden Layer 1	120	Sigmoid
Hidden Layer 2	60	Sigmoid
Dropout (0.5)
Hidden Layer 3	40	Relu
Hidden Layer 4	30	Relu
Dropout (0.5)
Hidden Layer 5	15	Relu
Output layer	1	Sigmoid
Learning rate: 0.0004
Loss function: Binary crossentropy
Optimizer: Adam
Metrics: Accuracy
Epoch number: 100
Batch size: 4

Model of Data Set 2a

Subject and decision (as binary) parts of freedom of expression–related cases take place in the Data Set 2a. The established model for this data set, which is named Model 2a, comprises six hidden layers in addition to input and output layers. For the hidden layers second and fifth, dropouts with the value of 0.5 have been applied. The input layer nodes number of the model is 1,865. Actually, the subject part of cases is not very broad. However, since the number of cases in the data set is a lot, the data set includes too many different words. Therefore, the number of words which also means the column number of document-term matrix is great enough. The hyperparameters values and model architecture are presented in Table 4. The Model 2a has been established relatively wider than Models 1a and 1b, as both input layer and data size are greater.

OPEN IN VIEWER

Table 4.

Model Architecture of the Model 2a.

	Number of Nodes	Activation Function
Input layer	1,865
Hidden Layer 1	30	Sigmoid
Hidden Layer 2	50	Sigmoid
Dropout (0.5)
Hidden Layer 3	150	Relu
Hidden Layer 4	250	Sigmoid
Hidden Layer 5	150	Relu
Dropout (0.5)
Hidden Layer 6	25	Sigmoid
Output layer	1	Sigmoid
Learning rate: 0.00025
Loss function: Binary crossentropy
Optimizer: Adam
Metrics: Accuracy
Epoch number: 120
Batch size: 8

Model of Data Set 2b

The reasoning and decision (as binary) parts of freedom of expression–related cases compose the Data Set 2b. The Model 2b, which has been developed for the Data Set 2b, contains four hidden layers and input, output layers as traditional. Also, dropouts with a value of 0.5 have been applied to the second and third hidden layers. The model has the greatest number of input layer nodes as 35,140 because this part of cases’ size is quite broad and has too many words like the Data Set 1b. As might be expected, the size of the document-term matrix is very large. Therefore, this model is the slowest running model compared to the others. The architecture and hyperparameters values of the developed Model 2b are given in Table 5.

OPEN IN VIEWER

Table 5.

Model Architecture of the Model 2b.

	Number of Nodes	Activation Function
Input layer	35,140
Hidden Layer 1	200	Sigmoid
Hidden Layer 2	150	Sigmoid
Dropout (0.5)
Hidden Layer 3	100	Relu
Dropout (0.5)
Hidden Layer 4	25	Sigmoid
Output layer	1	Sigmoid
Learning rate: 0.00025
Loss function: Binary crossentropy
Optimizer: Adam
Metrics: Accuracy
Epoch number: 35
Batch size: 8

Results

The models developed using the training data were tested on test data, and how successfully the model worked was presented using various performance criteria. Accuracy, precision, recall, F1 score, Cohen’s κ, and AUC performance criteria and calculation methods used in the study can be briefly explained through Confusion matrix (Table 6).

OPEN IN VIEWER

Table 6.

Confusion Matrix.

		Actual
		No Violation (0)	Violation (1)
Predicted	No violation (0)	A	B
	Violation (1)	C	D

Accuracy

This is the metric widely used to measure the success of a model. Accuracy value is calculated as the ratio of correct estimates to the total data set. However, it cannot be said that it is sufficient by itself, especially when evaluating unbalanced data sets.

Accuracy = A + D A + B + C + D .

Precision

It shows how many of the values predicted as “no-violation” are actually no-violation. The higher this value is, the better the model performs in classification.

Precision = A A + C .

Recall

It is a metric that shows how much of the data that are actually no-violation are estimated as no-violation. It is desired to be as high as possible.

Recall = A A + D .

F1 Score

This value is the harmonic mean of precision and recall values. It is an important measure of how well the model is performing and is often used in comparisons. It is more appropriate to use this value instead of accuracy, especially in models built on the basis of unbalanced data.

F 1 score = 2 × precision × recall precision+recall .

(Receiver Operating Characteristic [ROC]) AUC

It is one of the most important criteria used to evaluate the performance of a classification model. Like F1 score and Cohen’s κ, this criterion is also widely used when working with unbalanced data sets. AUC means “Area under the ROC curve.” The larger the area covered, the better the established model at classifying the given classes. In other words, it is an indicator of how much the classification made avoids misclassification.

Table 7 presents the mean results of the performance criteria of the test data sets. The mean values of test data sets were obtained by using models which are examined on each of the train and validation data sets. First of all, when TF-IDF and BoW word embeddings methods are evaluated, it can be argued that both methods can be used to achieve successful results in a prediction using MLP. However, we have observed that using the BoW method for data sets containing a large number of variables (word) is more efficient.

OPEN IN VIEWER

Table 7.

The Results of the Study.

Data Sets	Accuracy		Precision		Recall		F1 Score		Cohen’s κ		AUC
	BoW	TF-IDF	BoW	TF-IDF	BoW	TF-IDF	BoW	TF-IDF	BoW	TF-IDF	BoW	TF-IDF
Data Set 1a	0.852	0.873	1	1	0.825	0.850	0.903	0.918	0.603	0.644	0.965	0.937
Data Set 1b	0.978	0.936	0.976	0.932	1	1	0.987	0.964	0.908	0.691	1	1
Data Set 2a	0.810	0.820	0.815	0.790	0.729	0.805	0.769	0.795	0.609	0.635	0.871	0.898
Data Set 2b	0.943	0.923	0.961	0.964	0.933	0.890	0.946	0.925	0.886	0.846	0.968	0.966

Note. BoW = Bag of Words; TF-IDF = Term Frequency—Inverse Document Frequency.

When the results are evaluated in general, it is clear that each model developed for each data set exhibits highly successful performance and makes satisfactory level on correct prediction. The highest level of accuracy has been obtained in the model developed for Data Set 1b among the data sets using both TF-IDF method and the BoW method. Nevertheless, considering that the data sets used in the study are mostly unbalanced, it would be more appropriate to evaluate on the F1 score, Cohen’s κ, and AUC values. Among the data sets created by the BoW and TF-IDF method, the highest F1 score and Cohen’s κ values have been achieved in the model developed for data set b’s. Remember that data set b’s comprises reasoning and decision section of different cases and they include more variable (words) than data set a’s. Therefore, this result can be interpreted as coherent. It can be plausibly suggested that the most successful prediction is realized with the model developed using Data Set 1b. In other words, by using reasoning parts of public morality cases, more than 96% success has been achieved (F1 score) in the prediction of the decisions of the cases. Add to this, according to F1 scores, the achieved success rates in the prediction of the decisions of the cases are the following: average 90% for the public morality cases using subject parts, average 78% for the freedom of expression cases using subject parts, and average 93% for the freedom of expression cases by using reasoning parts.

According to the findings, the Cohen’s κ values of all models developed for the data sets in the study are greater than 60% or even 80%, demonstrating an almost perfect agreement. This means that the correct prediction is not by chance and is not due to an unbalanced data set. Furthermore, most of the AUC values above 90% indicate that the result has significantly avoided misclassification. When the F1 score, Cohen’s κ, and AUC values are considered together, it can be clearly maintained that the established models work correctly, and the prediction is reliable even though the data sets are unbalanced.

As explained in the second section, there are many studies (Aletras et al., 2016; Ashley & Brüninghaus, 2009; Barros et al., 2018; Bhilare et al., 2019; Katz et al., 2017; Liu & Chen, 2017; Medvedeva et al., 2019; Ruger et al., 2004; Shaikh et al., 2020; Sulea et al., 2017) that have achieved successful results by applying different statistics and ML methods in order to predict court decisions. What distinguishes this investigation from these comparable studies is that the high-accuracy prediction has been achieved by using only a small data set. Furthermore, the subject part of the case constituting only a very small part of the data that have brought about the highest accuracy in the study. Normally, it is expected that prediction with the MLP by using the reasoning section of cases would be more successful since this section embodies important clues about the outcome of a case. However, the results of our analysis show that the subject sections of cases can be successfully used for the prediction with the MLP. In other words, the predictions derived from the subject section are as successful as reasoning section of cases in terms of performance. This has been achieved by building an appropriate DL model.

Conclusion

The research examines decisions arising out of individual applications issued by the TCC between 2012 and June 2020 on the subjects of public morality and freedom of expression, which are taken as data. We seek to predict whether an application regarding these issues has resulted in a violation or a nonviolation or inadmissible decision. In the first data sets, the results of the cases are predicted using only the subject part of the cases, and in the second data sets, only the reasoning part of the cases has been employed to predict the outcomes of the cases. We have attained satisfactory success rates on predictions with the MLP by using the subject or reasoning sections of texts derived from well-structured court rulings. The results of our model testing have shown that the textual content of the “subject” and “reasoning” sections predict quite well the outcome of the case. Obviously, predicting the results of cases using the reasoning part is not a particularly challenging task as it comes about almost simultaneously with the judgment. Our principal goal in including the reasoning section is to validate and verify the prediction solely based on the subject part. The main finding of the study is that the outcome of the cases has been predicted with a very high success by utilizing only the subject part of the cases. This prediction has been almost as successful as the predictions based on the reasoning part. We can plausible suggest that the developed model has successfully predicted what will be the outcome of a case by using only the subject part of the individual applications related to public morality and freedom of expression. This is important as it also illustrates that it is possible to make accurate predictions on legal texts in Turkish language with DL techniques. Add to this, the study demonstrates that even with relatively small data sets, high estimation performance can be achieved by applying basic DL methods with appropriate modeling.

On the other hand, we must be caution against reading too much into the results of the research. Although accurate predictions of the applications based on NLP techniques can be made, their connotations for procedural rights should not be overlooked. Currently, available techniques should not be considered as a substitute for judicial decision making capabilities of human mind. These techniques can be utilized as a part of decision support systems accelerating judicial proceedings. Within this context, the model developed in our research could be effectively applied in prioritizing individual complaints indicating a high likelihood of a violation ruling. In this way, the TCC can reduce the backlogs of individual applications piled up on the court. Regarding future studies, they may be undertaken by employing more complex word vector methods such as “Word2Vec” or “GloVe” and/or different DL methods such as recurrent neural networks using larger data sets.

Supplemental Material