Solving arithmetic word problems: A deep learning based approach

The proposed work contributes in various aspects of solving word problems as given in the list below.

A bi-directional LSTM based deep learning technique to classify operations

We propose a simple approach to form the final equation without using any templates as used in some existing DL based math solvers. Additionally, the code will be published after the publication of the paper. Next, in Section 2, we describe some specifically related works in solving arithmetic word problems. Later we explain our system in details with all important components in Section 3. Section 4 contains a description about datasets, dataset variants, and results with critical discussions followed by some conclusions in Section 5.

We adapted and modified the concept of an object-oriented modeling approach of arithmetic word problems from the work of [19] which reported entity role labeling for math word problems based on Semantic Role Labelling (SRL) and FrameNet. Our work requires the extraction of the desired pieces of information to identify global cue (G). The global cue G –{‘owner’, ‘item’, ‘attribute’, ‘location’, ‘verb’} are extracted using a rule-based approach similar to [20].

Preprocessing: A rule-based preprocessing module is integrated with Knowledge Base with two sub-components. One is to substitute coreferences if any (i.e., ‘he’, ‘she’, ‘his’, ‘her’, ‘it’, etc.) and another is to eliminate conjunctions such as ‘and’, ‘but’, ‘,’, etc., to simplify and rephrase an input problems. This is also used to prepare the different variants of datasets as in Section 4.

Various case studies are given in Table 2 based on word problem examples to demonstrate preprocessing, IE tasks related to different system components such as G, Q, μ, κ, etc.

The BiLSTM architecture [7] consists of a set of recurrently connected network modules, known as memory blocks or LSTM cells. In this work, we used bi-directional memory blocks to classify the four basic operations. Unlike the state-of-the-art equation template-based method  [15, 42], we propose a template-free framework for the operation classification task.

Firstly, the word problem is converted into a vector using pre-trained word vectors. We considered an input dimension of 1000. Widely used Glove word embedding [25] is used to generate the sequence of input from the word problem (M). Given an input sequence x = x₁ . . . x _T , a standard BiLSTM classifier computes the hidden vector sequence h = h₁ . . . h _T and the output class y. The model is defined using the following equations.

h i → = LSTM fw ( h → i - 1 , x i ) ∥(8)

h i ← = LSTM bw ( h i + 1 ← , x i )(9)

δ = drop ( h i → , h i ← )(10)

y = softmax ( [ h i → , h i ← ] )(11)

We used T + 100 layer model is used for the classification task following the state-of-the-art BiLSTM reported in [7]. Finally, we added a 4 class classifier as a softmax layer is added to classify the operation.

We also explored two other deep learning frameworks, Convolutional Neural Network (CNN) [37] and Hierarchical Attention Network (HAN) [40], for solving arithmetic word problems and to compare them with our proposed BiLSTM based method. We used a 3 layered convolutional architecture consisting of 128 filters (size 5) and max-pooling of 5 × 5. A HAN consisting of a hierarchically connected time distribution layer and an LSTM layer is also experimented along with the word embedding layer. In each case, Rectified Linear Unit (ReLU) is used as the activation function.

Irrelevant information removal by identifying relevant quantities is one of the important tasks in solving arithmetic MWPs. We developed IIRU, a relevance classifier, and propose a set theory-based approach to find the desired cues from the question sentence and the relevant micro statements (s _i ) of a word problem. We categorized word problems as discussed by analyzing various word problem categories proposed by various researchers in the literature [6, 27] and set miscellaneous rules to identify them categorically. Table 2 shows one example from each of the categories. The problems belonging to these categories have some basic differences in characteristics.

Initially, the word problems are categorized as - Change (Cha), Compare (Com), Combine (Cob), and Division-Multiplication (Div-Mul), as in Equation 12. The set of quantities, Q is extracted from M as defined in equation 6.

category = { Cha , Com , Cob , Div - Mul }(12)

Next, we describe the generic procedure to identify the relevant quantities (RQ) for all the categories (cf. Table 2 with examples from each category). Firstly, the knowledge base (β) is taken as input. We define the rules to identify the relevant micro statements as in Equations  13 and 14, where at first Equation  13 is tested and if it fails to retrieve at least two such statements, each with a single quantity, the second condition presented in Equation 14 is tested to retrieve two quantities according to a set of precedence. We set precedence as {location, attribute, item, owner} to match (γ ! = ϕ) (cf. Table 2) and find at least two quantities from the word problem following the (γ) in Equation 14. In Equation  13, R and I denote the relevant and irrelevant information (quantities) respectively. class ( μ i ) = { R κ = { ϕ } I Else(13)

match ( γ ) = μ i ⋂ μ q(14)

Some category-specific and problem-specific rules are also applied to identify relevant quantities for different categories. E.g., we separately set up rules to identify relevant quantities for ‘money word problems’ (consisting of ‘$’ or dollar in case of our dataset) by grouping the verbs related to monetary transaction sense (using VerbNet) such as ‘pay’, ‘spend’, ‘buy’, ‘purchase’, etc. We select the corresponding micro statements containing these verbs as relevant micro statements. For Div-Mul type problems, owner is not considered as part of G, since it has no importance to identify relevant quantities as observed in the datasets (cf. last example of Table 2). It is to be noted that if a word problem contains only 2 quantities, these quantities are automatically relevant, otherwise, the word problem is incomplete, as noticed in the datasets.

Answer generation is the final task in the pipeline. This task combines the predicted operation and the relevant quantities to form the desired equation (cf. Equation 1) which is further evaluated to generate the answer to the input word problem. Algorithm 3 presents the pipeline of the proposed DLAWPS.

4.1 Datasets

[14] proposed a framework which can check grammatical errors, reduces arithmetic MWP datasets (or extend them) by minimizing template overlaps and lexical overlaps among the word problems. [14] also categorized all such datasets (cf Table 1 of [14]) with characteristically similar word problems to motivate researchers to solve any one or more of such arithmetic MWP datasets. We carried out our experiments on two such datasets available in the MAWPS [14] word problems repository –AddSub (MWP _AddSub ) dataset consisting of a reduced addition-subtraction word problems [9] and SingleOp (MWP _SingleOp ) [27] dataset consisting of single operation word problems. [14]) published a reduced AddSub dataset which is reduced in the sense of numbers of problems, not in the sense of types of problems. They eliminated similar (redundant) problems examples of which are given below. In the original AddSub dataset, we found two such problems.

“There are 7 crayons in the drawer. Mary took 3 crayons out of the drawer. How many crayons are there now?”

These datasets consist of MWPs that require single equation with single operation in order to be solved; this inspired us to formulate the single final equation as in Equation 1. We combined these 2 datasets into a Combined dataset, i.e., MWP _Combined  = MWP _AddSub  ∪ MWP _SingleOp , with 917 word problems comprising of all four basic operations.

We prepared 4 dataset variants (DV) for each of the 3 datasets (MWP _Combined , MWP _AddSub , MWP _SingleOp ) as given below, thus giving rise to a total of 12 datasets. (1) DV1: Actual dataset, i.e., the standard dataset in its original form. (2) DV2: Modified dataset after eliminating the conjunctions. (3) DV3: Modified dataset after complete preprocessing. (4) DV4: Rephrased word problems after eliminating sentences with irrelevant information after complete preprocessing. The main motivation of creating the variants of the dataset is to understand the effectiveness of CNN, HAN and proposed BiLSTM on the different representations of the word problems. We carried out our experiments using them on all (12) dataset variants. It is to be noted that IIRU is used only as a preprocessing step to prepare the DV4 dataset variants in which sentences with irrelevant information or quantities are removed to rephrase the word problems.

4.2 Experiments and results

All the experiments were evaluated using a 10-fold cross validation framework taking validation size 20%. We used categorical cross entropy loss, RMSprop optimizer, batch size 2, word embedding vector of 100 dimension, and used 40 epochs in each case. Table 3 summarizes the results of our experiments with RNN, CNN and HAN, on each of the 4 variants (DV1, DV2, DV3 and DV4) for each of the 3 datasets (AddSub, SingleOp and Combined). Dataset variant-specific best scores are shown in Table 3 in italics and the dataset-specific best scores are shown in bold. The overall best score is underlined in Table 3. It can be observed from the results that our RNN based BiLSTM method performs better than CNN and HAN on most of the dataset variants. It can also be observed from the results in Table 3 that the DV1 is the most effective dataset for solving word problems for AddSub and SingleOp, while DV2 and DV3 for the Combined dataset produce better accuracies than DV1. The performances on DV4 are mixed, however, DV4 performances are steadier (for proposed BiLSTM) than the other variants. It was expected that on DV4 the system should produce better results, but eventually, that does not happen. Possibly, since the structure of the input problems is changed due to preprocessing, it may lead to some information loss when training the models. Also, preprocessing is not completely accurate for all cases.

Figure  3 presents the training and validation accuracy and Fig.  4 presents the loss, both on the DV1 variant of the Combined dataset for the BiLSTM model. It is observed that the proposed network does not over-fit with the limited amount of training data. Figure  5 presents the confusion matrix for classifying the four operations (‘+’,‘-’,‘*’, and ‘/’).

OPEN IN VIEWER

Fig. 2

Framework of the arithmetic MWP operation classifier consisting of BiLSTM with 4 softmax layer to classify operation (‘+’, ‘-’, ‘*’, and ‘/’).

OPEN IN VIEWER

Fig. 3

Training and validation accuracy of the proposed architecture for DV1 of Combined dataset.

OPEN IN VIEWER

Fig. 4

Training and validation loss of the proposed architecture for DV1 of Combined dataset.

OPEN IN VIEWER

Fig. 5

Confusion matrix of the proposed operation classifier for DV1 of the Combined dataset.

It was noticed that 17.34% of the word problems (i.e., 159 out of 917 problems) in the Combined dataset contain irrelevant quantities. It is obvious that, without IIRU, we will not get correct results for any of these 159 problems which will reduce the accuracy to 77.43% from 94.77% for Combined - DV1 (cf. Table 3). This is true for all variants. Therefore, the identification of such quantities is important. Figure  6 presents the confusion matrix for our relevancy classifier, IIRU, where the labels ‘R’ and ‘I’ refer to Equation 13.

Table 4 presents the performance comparison of our system with respect to other similar systems, on the AddSub and SingleOp datasets. Table 4 clearly indicates that our system outperforms the current state-of-the art systems on the same datasets (as available in the literature.) Our BiLSTM based system provides accuracy of 94.08% and 97.15% on the AddSub and SingleOp datasets, respectively, on which the previous state-of-the-art results were 86.07% [21] and 79.5% [17]. It is to be noted that we used a reduced AddSub dataset consisting of 355 problems instead of 395 problems in the AddSub dataset on which the results of other systems are reported. All the results on SingleOp dataset are on 562 problems for all the systems including ours.

The existing state-of-the-art systems generally used supervised learning approach to learn various system components such as equation template [15], verb categories [9], equation tree [27, 29], equation formulation [21], etc. [17] used a rule-based approach to identify the desired operation and used Tags (cf. Section 2) to identify the relevant quantities. Therefore, our system is critically different as we used deep learning based approach which helps us get rid of manual feature engineering. [27] used supervised relevance classifier trained on a dataset with very few problems with irrelevant quantities. Rather, we used a more realistic approach to identify the relevant quantities (cf. Section 2). [35] used a reinforcement learning approach to classify the operations and tested their performance on the same datasets.

Our system generates outstanding results and beats other similar systems mainly for two reasons as given below.

To the best of our knowledge, this is the first use of BiLSTM in classifying operations in solving word problems from AddSub and SingleOp datasets and the BiLSTM model outperforms other state-of-the-art systems.

OPEN IN VIEWER

Fig. 6

Confusion matrix of the proposed relevance classifier (IIRU) for DV1 of the Combined dataset.

Our system resulted in 47 errors overall on the Combined actual (i.e., DV1) dataset, out of which 24 errors are in the AddSub dataset and 23 errors are in the SingleOp dataset. The sources of errors are given below.

Classification error: The system predicts wrong operations for 18 (5 AddSub + 13 SingleOp) word problems.