HFD: Hierarchical feature decoupling for SQL generation from text

Abstract

Text-to-SQL, a computational linguistics task, seeks to facilitate the conversion of natural language queries into SQL queries. Recent methodologies have leveraged the concept of slot-filling in conjunction with predetermined SQL templates to effectively bridge the semantic gap between natural language questions and structured database queries, achieving commendable performance by harnessing the power of multi-task learning. However, employing identical features across diverse tasks is an ill-suited practice, fraught with inherent drawbacks. Firstly, based on our observation, there are clear boundaries in the natural language corresponding to SELECT and WHERE clauses. Secondly, the exclusive features integral to each subtask are inadequately emphasized and underutilized, thereby hampering the acquisition of discriminative features for each specific subtask. In an endeavor to rectify these issues, the present work introduces an innovative approach: the hierarchical feature decoupling model for SQL query generation from natural language. This novel approach involves the deliberate separation of features pertaining to subtasks within both SELECT and WHERE clauses, further dissociating these features at the subtask level to foster better model performance. Empirical results derived from experiments conducted on the WikiSQL benchmark dataset reveal the superiority of the proposed approach over several state-of-the-art baseline methods in the context of text-to-SQL query generation.

Keywords

Text-to-SQL multi-task learning discriminative features feature decoupling

1. Introduction

Relational databases, repositories of extensive data, persist as inaccessible inaccessible to non-technical stakeholders who lack proficiency in Structured Query Language (SQL) and the foundational database schema. Over the course of the preceding decades, the realm of text-to-SQL, which endeavors to transmute natural language queries into SQL equivalents, has experienced an ascending wave of scholarly. This surge in interest is notably propelled by the contemporary strides made in the domain of deep neural networks and the amassment of substantial corpora, thereby yielding enhanced solutions for the text-to-SQL paradigm. In view of the intrinsic diversity in the nature of inquiries that may traverse either a solitary table or multiple tables, the text-to-SQL process can be broadly classified into two distinguishable categories: single-table-based and multiple-table-based. Within the purview of this paper, our particular focus is on the single-table-based text-to-SQL approach.

Table 1
Contents in database table 1-10416547-1

Lyricist	Music genre/style	Major instrument (s)	Lyrics theme/style	Duration
…	…	…	…	…
Huang jun lang	Hip-hop, orchestra	Violin, cello, piano, synthesizer	Detective story	3:46
Zeng yu ting	r&b	Piano, synthesizer	Romance	3:49
Vincent fang	r&b	Guitar (acoustic)	Romance	3:50
Vincent fang	Hip-hop, tango	Violin, guitar (acoustic)	Friendship	3:50
Jay chou	Hip hop soul	Violin, cello, guitar (acoustic)	Ffamily, domestic violence	3:51
Liu geng hong	r&b	Guitar (acoustic)	Romance	3:51
Huang jun lang	Hip-hop, orchestra	Orchestra	Motivational	3:51
…	…	…	…	…

In the realm of single-table-based text-to-SQL models, a classification can be made into three distinct categories: sequence-to-sequence based, syntax-based, and sketch-slot-filling based methodologies. The sequence-to-sequence based models operate by generating a sequence of SQL tokens and schema elements, ultimately culminating in the production of the final SQL query prediction or a substantial portion thereof. In essence, they endeavor to undertake the conversion of the input natural language sequence into a SQL sequence. Notably, an early pioneer in the text-to-SQL domain, known as Seq2SQL [43], employed a sequence-to-sequence paradigm. However, it failed to fully account for the rigorous syntactic constraints inherent to SQL, leading to the ineluctable occurrence of syntactically flawed queries during the prediction phase. Subsequent to the sequence-to-sequence paradigm, the syntax-based approaches emerged as an evolution, which generate a sequence of syntax rules rather than mere tokens. These syntax rules manifest as clauses, and upon their application, conduce to the formation of a valid SQL query [42, 5, 9, 2, 25, 32]. An inherent advantage of syntax-based techniques is their ability to substantially mitigate the likelihood of generating erroneous tokens or syntactically flawed queries. Nonetheless, these methods exhibit diminished recall and necessitate substantial human intervention in formulating the pertinent rules, a process inherently challenging in its own right. Concurrently, the sketch-slot-filling based approaches endeavor to transmute the intricate endeavor of SQL query generation into a more tractable task, focusing on predicting specific segments within queries (e.g., determining which table columns will populate the SELECT clause). This paradigm effectively reimagines SQL generation as a classification task with accompanying methodologies, as witnessed in the methodological works of [37, 41, 11, 10, 17, 18, 38, 3]. A body of research has conclusively demonstrated the efficacy of sketch-slot-filling based approaches in handling single-table-based text-to-SQL challenges, yielding markedly superior results when juxtaposed with their aforementioned counterparts.

For the single-table-based text-to-SQL, every SQL query is composed of a “SELECT” (column/aggregation) clause with a “WHERE” (condition) clause that consists of one or multiple conditions. It can be observed from the WikiSQL dataset that there is a clear decoupling of natural language questions corresponding to SQL queries. For example, as shown in Fig. 1, “What is the total number of lyricist” clearly corresponds to the SELECT clause, while “where the lyrics theme is romance and the song lasts 3:50” corresponds to the WHERE clause. Then, turn attention to slot-fillings in SELECT and WHERE clauses where each slot can correspond to a specific word or feature in the natural language question. Herein, “Lyricist” clearly corresponds to the column in the SELECT clause. Table 1 shows the table schema corresponding to the question in Fig. 1. The feature representation acquired by the slot-filling-based approach is slightly useless or even negative for individual subtasks.

Figure 1.

Example diagram of hierarchical feature decoupling.

Moreover, Qin et al. [22] found that some words or features associated with some class labels usually exist in text classification, but there are also many other common features that cannot distinguish different classes. They proposed a feature decoupling method to enhance the feature representation and make it more discriminative in classification. In addition, a series of work in computer vision, acoustics, and other areas where representation learning are performed by decoupling [34, 35, 27, 14].

Based on our observations from the dataset and the problems with current slot-filling-based methods, we propose a text-to-SQL approach with hierarchical feature decoupling. Specifically, we designed a hierarchical feature decoupling model based on HydraNet [17]. The model achieves a two-part feature decoupling for the subtasks in SELECT and WHERE clauses. Decoupled features are employed for slot prediction in corresponding clauses. In slot prediction, we draw on the work [28] to achieve feature decoupling between subtasks.

We conclude our contributions as follows.

•

Based on our observations from the dataset and the problems with current slot-filling-based methods, we propose a text-to-SQL approach with hierarchical feature decoupling.

•

The validity of the model was verified on the benchmark dataset WikiSQL, and the proposed model achieves better performance than several state-of-the-art baselines. Our source code will be publicly available on GitHub.2

https://github.com/zhangxu90s/HFD.

The remainder of this paper is structured as follows: Section 2 succinctly reviews pertinent prior work, Section 3 offers an exposition of the HFD model’s implementation, Section 4 expounds upon the conducted experiments and their corresponding outcomes, and, lastly, Section 5 delivers a comprehensive denouement to this paper.

2. Related work

2.1 Natural language to structured query language

In the domain of Text-to-SQL, the formalization of the concept was preceded by earlier developmental efforts. Norouzifard et al. [20] introduced an algorithm that enabled the selection of a query statement from a set of multiple query statements by parsing and subsequently converting the input statement into an SQL query statement. This marked an initial step in advancing the field. Rao et al. [23] later put forth an architectural framework aimed at transforming queries into SQL statements through a consideration of semantics and syntax. Subsequent research further refined the text-to-SQL process. Chaudhari [21] proposed a system with the capability to handle both simple queries and aggregate functions, broadening the scope of its applicability. Ghosh et al. [8] made strides in developing an automatic query generation system that could support more complex multi-table queries. Reinaldha and Widagdo [24] employed semantic rules to establish mapping relationships between words in a question and attributes in a database, further enhancing the precision of text-to-SQL conversion. Singh and Solanki [26] adopted an approach that utilized verb lists, noun lists, and predefined rules to facilitate the mapping of attributes and tables to words within sentences, even in cases of ambiguous textual input. Kate et al. [12] introduced a model designed to transform natural language problems into SQL queries, contributing to the evolution of text-to-SQL methodologies. Utama et al. [30] presented the DBPal system, an innovative solution that harnessed deep learning to convert natural language statements into SQL, effectively allowing users to interact with databases without needing a detailed understanding of the database schema or query characteristics.

Recent advancements in the field of text-to-SQL have seen a shift towards the utilization of large pre-trained models such as BERT [13], XLNet [40] as natural language problem solvers and feature extractors for database tables, resulting in substantial enhancements in model effectiveness. Hwang et al. [11] introduced the BERT-based network model SQLova, and He et al. proposed the MT-DNN [15] based network model X-SQL, drawing inspiration from the BERT-style architecture. Furthermore, Microsoft Dynamics365AI presented X-SQL [10] in 2019, and in 2020, Microsoft introduced the HydraNet model [17], both of which are based on pre-trained models. Building upon the progress of the HydraNet model [17], our work in this paper aligns with the HydraNet model and puts forth a novel approach to text-to-SQL conversion that leverages slot-filling techniques, contributing to the ongoing evolution of text-to-SQL methodologies.

2.2 Multi-task learning

Multi-task learning (MTL) stands as a pivotal paradigm within the realm of machine learning, manifesting its utility by harnessing knowledge from one or more source tasks to enhance the performance of a designated target task. The overarching objective of MTL is to fortify the learning process by encapsulating the modeling of multiple tasks within a singular computational framework. Notably, the inception of a deep multi-task learning network with hard parameter sharing can be attributed to Caruana et al. [1]. This methodology for MTL pertains to the concurrent training of multiple tasks while internally intertwining their performance metrics. This approach renders the data derived from supplementary tasks as a form of induced error, serving as a constraint that fosters advancements in overall accuracy and learning efficiency.

Of particular relevance, hard parameter sharing stands as a preeminent MTL technique within neural networks, prominently addressing concerns of overfitting, as elucidated by Thung et al. [29]. Various avenues for enhancing the efficacy of deep multi-task learning methodologies have been explored. A noteworthy innovation encompasses the cross-stitch network, an end-to-end trainable architecture that offers generalizability across multiple tasks, as advanced by Misra et al. [19]. In the context of convolutional neural networks, Fang et al. have introduced a dynamic multi-task learning approach characterized by the Task Transfer Connection (TTC) framework [7]. This innovative structure introduces TTC factors between analogous network layers pertaining to multiple tasks, thereby facilitating the sharing of knowledge among these tasks. Yang and Hospedales [39] have proposed a novel tensor decomposition technique within the domain of deep multi-task learning, grounded in matrix decomposition principles and seamlessly amalgamating tensor decomposition methods with multi-task learning strategies. Moving forward, the work of Chen et al. [4] has bolstered the facet of multi-task learning from a model training perspective, through the presentation of a gradient normalization algorithm engineered to adapt deep multi-task learning networks and optimize adaptive loss balancing. Furthermore, the research by Tang et al. [28] has introduced a pioneering MTL model, denoted as Progressive Layered Extraction (PLE), which explicitly segregates task-sharing and task-specific parameters. This demarcation mitigates negative transfer and the see-saw phenomenon, thereby fostering more efficient information exchange and joint representation learning. In the context of slot prediction, insights from Tang et al. [28] have been harnessed to achieve feature decoupling across subtasks.

3. Hierarchical feature decoupling model

We present a novel neural network model characterized by hierarchical feature decoupling, as depicted in Fig. 2. The framework’s application is directed toward the text-to-SQL task, a computational architecture composed of four integral modules, namely: a semantic representation module, a clause feature decoupling module, a slot feature decoupling module, and a slot prediction module.

Figure 2.

Architecture of hierarchical feature decoupling model.

In the ensuing sections, we embark upon a comprehensive exposition of these components. Initially, we expound upon the intricacies of semantic representation, encompassing the encoding of natural language inquiries and the tabular schema. Subsequently, we delve into the intricate details of the clause feature decoupling module. This is succeeded by an exhaustive examination of the slot feature decoupling module, which specializes in the isolation of individual slot features within clauses. Finally, we furnish a detailed discourse on the slot prediction module, dedicated to the anticipation of the content resident within corresponding slots.

3.1 Semantic representation

In our formalization, we represent a natural language query as $Q=<q_{1},\ldots,q_{n}>$ , wherein ‘n’ signifies the query length and $q_{i}$ corresponds to the ‘i-th’ token within the query. To encapsulate the structural elements inherent to the query, we employ a schema representation, denoted as $S=\linebreak<s^{1}_{1},s^{2}_{1},\ldots,s^{1}_{m},s^{2}_{m},\ldots>$ , where $s^{j}_{i}$ signifies the ‘j-th’ token within the ‘i-th’ header, and ‘m’ represents the total count of headers. We adopt the RoBERTa model [16] for the encoding of both the question Q and the headers S:

$\displaystyle E=[\mathrm{CLS}],q_{1},\ldots,q_{n},[\mathrm{SEP}],s_{1}^{1},s_{% 1}^{2},\ldots,[\mathrm{SEP}],s_{m}^{1},s_{m}^{2}\ldots[\mathrm{SEP}]$ (1)

3.2 Clause feature decoupling

In this study, a dual multi-head self-attention mechanism was employed for the purpose of feature extraction, drawing upon the seminal work by Vaswani et al. [31]. It should be noted that the parameters of these two distinct networks were not shared, a crucial design choice. This segregation of parameters allowed for the specific exploration of semantic representations within the SELECT and WHERE clauses of single-table-based SQL queries. Consequently, these two networks were found to facilitate the decoupling of the SELECT and WHERE clauses, and the application of multi-head self-attention was faithfully replicated as presented below.

$\displaystyle\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac% {QK^{\top}}{\sqrt{d_{k}}}\right)V$ (2)

where $\bm{Q}\in\mathbb{R}^{n\times d_{k}},\bm{K}\in\mathbb{R}^{m\times d_{k}},\bm{V}% \in\mathbb{R}^{m\times d_{v}}$ , they are matrix of three $n\times d_{k},d_{k}\times m,m\times d_{v}$ dimensions.

$\displaystyle\text{head}_{i}=\text{Attention}(\bm{Q_{i}}\bm{W}_{i}^{Q},\bm{K_{% i}}\bm{W}_{i}^{K},\bm{V_{i}}\bm{W}_{i}^{V})$ (3)

where $\bm{W}_{i}^{Q}$ , $\bm{W}_{i}^{K}$ and $\bm{W}_{i}^{V}$ represents the parameter matrix.

In accordance with the fundamental principles of the attention mechanism, the vectors denoted as $Q$ , $K$ , and $V$ are systematically transformed via a parameter matrix, following which they undergo encoding through the application of attention, as exemplified in Eq. (2), to yield the constituent $\text{head}_{i}$ components, as explicitly delineated in Eq. (3). This process is iteratively executed for a total of $h$ repetitions, culminating in the amalgamation of the resultant outcomes into a cohesive sequence, denoted as $\operatorname{MultiHead}(\bm{Q},\bm{K},\bm{V})$ , as depicted in the subsequent equation4.

$\displaystyle\operatorname{MultiHead}(\bm{Q},\bm{K},\bm{V})=\operatorname{% concat}(\text{head }_{1},\ldots,\text{head}_{h})$ (4)

In this way, we obtain two decoupled feature representations, $C_{\textit{select}}$ and $C_{\textit{where}}$ .

SELECT clause.

$\displaystyle C_{\textit{select}}=\operatorname{MultiHead}(\bm{E},\bm{E},\bm{E})$ (5)

WHERE clause.

$\displaystyle C_{\textit{where}}=\operatorname{MultiHead}(\bm{E},\bm{E},\bm{E})$ (6)

3.3 Slot feature decoupling

In Section 3.2, we derive the decoupled features denoted as $C_{\textit{select}}$ and $C_{\textit{where}}$ , facilitating their application in characterizing the slots within their respective clauses. The separation of SELECT and WHERE clauses is evident in both SQL queries and natural language interrogations. Nevertheless, tasks pertaining to slot prediction within the SELECT and WHERE clauses are also treated independently. It is important to underscore that weak correlations or conflicts among individual slot predictions may impede the model’s capacity to effectively learn the associated parameters.

It is worth mentioning that a more efficient network architecture for achieving effective decoupling, referred to as the customized gate control model, has been previously proposed by Tang et al. [28]. We incorporate their methodology into our current investigation with the aim of attaining a decoupled feature representation of slot fillings.

As illustrated in Fig. 2, the network module within the present Section comprises multiple expert modules and task-specific tower networks. Each expert module encompasses numerous sub-networks, known as experts, with the number of experts in each module being a tunable hyper-parameter. Similarly, a tower network is a multi-layered network with width and depth as configurable hyper-parameters. In particular, shared experts are responsible for capturing common patterns, whereas task-specific patterns are extracted by dedicated task-specific experts. Each tower network assimilates knowledge from both the shared experts and its task-specific counterparts. This implies that the parameters of the shared experts are influenced by all tasks, while the parameters of the task-specific experts are exclusively shaped by their respective tasks.

The integration of shared experts and task-specific experts for selective fusion is accomplished through a gated network structure. The architecture of the gated network is based on a single-layer feed-forward network, employing SoftMax as the activation function, and the input serves as a selector that calculates the weighted sum of the selection vectors, representing the output of the expert. More precisely, the output of the gated network for task $k$ is mathematically expressed as:

$\displaystyle g^{k}(x)=\operatorname{Softmax}(W_{g}^{k}x)S^{k}(x)$ (7)

In the context of this study, the variable denoted as $x$ corresponds to the input representation, where $W_{g}^{k}\in R^{(m_{k}+m_{s})\times d}$ represents a parameter matrix. Here, $m_{s}$ signifies the count of shared experts, while $m_{k}$ pertains to the number of specific experts associated with task $k$ . The parameter d signifies the dimensionality of the input representation. Notably, $S^{k}(x)$ is a composite matrix, which comprises all vectors that have been selected, encompassing both the shared experts and those experts that are specific to task $k$ :

$\displaystyle S^{k}(x)=[E_{(k,1)}^{T},E_{(k,2)}^{T},\ldots,E_{(k,m_{k})}^{T},E% _{(s,1)}^{T},E_{(s,2)}^{T},\ldots,E_{(s,m_{s})}^{T}]^{T}$ (8)

The discrete realization of the particular implementation exhibits a bifurcation into two distinct segments, namely the SELECT clause and the WHERE clause. Employing the formalism denoted as Eq. (5), we are able to derive the disentangled slot features intrinsic to both the SELECT and WHERE clauses, encompassing entities denoted as $S_{\textit{col}}$ , $S_{\textit{agg}}$ , $W_{\textit{col}}$ , $W_{op}$ , $W_{no}$ , $W_{\textit{start}}$ , and $W_{\textit{end}}$ , as delineated in Eqs (3.3)–(15).

SELECT clause.

$\displaystyle S_{col}=\operatorname{Softmax}(W_{g}^{S_{\textit{col}}}C_{% \textit{select}})S^{S_{\textit{col}}}C_{\textit{select}}$ (9) $\displaystyle S_{\textit{agg}}=\operatorname{Softmax}(W_{g}^{S_{\textit{agg}}}% C_{\textit{select}})S^{S_{\textit{agg}}}C_{\textit{select}}$

WHERE clause.

$\displaystyle W_{\textit{col}}=\operatorname{Softmax}(W_{g}^{W_{\textit{col}}}% C_{\text{where}})S^{W_{\textit{col}}}C_{\text{where}}$ $\displaystyle W_{op}=\operatorname{Softmax}(W_{g}^{W_{op}}C_{\text{where}})S^{% W_{op}}C_{\text{where}}$ $\displaystyle W_{no}=\operatorname{Softmax}(W_{g}^{W_{no}}C_{\textit{where}})S% ^{W_{no}}C_{\text{where}}$ (10) $\displaystyle W_{\text{start}}=\operatorname{Softmax}(W_{g}^{W_{\text{start}}}% C_{\text{where}})S^{W_{\text{start}}}C_{\text{where}}$ $\displaystyle W_{\text{end}}=\operatorname{Softmax}(W_{g}^{W_{\text{end}}}C_{% \text{where}})S^{W_{\text{end}}}C_{\text{where}}$

where $W_{g}^{S_{\textit{col}}}$ , $W_{g}^{S_{\textit{agg}}}$ , $W_{g}^{W_{\textit{col}}}$ , $W_{g}^{W_{op}}$ , $W_{g}^{W_{no}}$ , $W_{g}^{W_{\text{start}}}$ , and $W_{g}^{W_{\text{end}}}$ represent the parameter matrix.

3.4 Slot prediction

The SQL prognostication module, as delineated in Lyu et al.’s work [17], is comprised of an assortment of sub-modules designed to autonomously anticipate discrete segments of SQL statements. An all-encompassing SQL statement can be dissected into two fundamental constituents, namely the Select Clause and the Where Clause. As expounded upon in Section 3.3, the feature representation vectors corresponding to the disentangled constituent elements, denoted as $S_{\textit{col}}$ , $S_{\textit{agg}}$ , $W_{\textit{col}}$ , $W_{op}$ , $W_{no}$ , $W_{\textit{start}}$ , and $W_{\textit{end}}$ , are acquired and scrutinized.

SELECT Clause. The S-col module is designed with the objective of identifying the column that corresponds to the specific inquiry and schema, while the S-agg module is designed to ascertain the appropriate aggregation operator for the given context.

$\displaystyle p_{s_{\textit{col}}}=\sigma(\mathbf{W}_{s_{\textit{col}}}\tanh S% _{\textit{col}})$ (11)

where the ${\mathbf{W}_{s_{\textit{col}}}}$ is parameter matrix, $\sigma$ is the sigmoid function.

The S-agg module endeavors to determine an optimal aggregation operator from a set of six potential selections, namely: {None, Max, Min, Count, Sum, Avg}.

$\displaystyle p_{s_{\textit{agg}}}=\operatorname{Softmax}(\mathbf{W}_{s_{% \textit{agg}}}\tanh S_{\textit{agg}})$ (12)

where $\mathbf{W}_{s_{\textit{agg}}}$ represents the parameter matrix.

WHERE Clause. In a manner analogous to the $S_{\textit{col}}$ approach, the $W_{\textit{col}}$ module undertakes the prognostication of column information through the utilization of vector representations:

$\displaystyle p_{w_{\textit{col}}}=\sigma(\mathbf{W}_{w_{\textit{col}}}\tanh W% _{\textit{col}})$ (13)

where $\mathbf{W}_{w_{\textit{col}}}$ represents the parameter matrix.

Moreover, it is imperative to elucidate that the W-operator (W-op) presents a trinary selection, encompassing the options $\{=,>,<\}$ . In addition, the W-value (W-val) module undertakes the task of ascertaining the conditional criteria through a process that entails the identification of the initiation and termination markers within the vector denoted as $W_{\textit{start}}$ and $W_{\textit{end}}$ .

$\displaystyle p_{w_{op}}=\operatorname{Softmax}(\mathbf{W}_{w_{op}}\tanh W_{op})$ (14)

where $\mathbf{W}_{w_{op}}$ represents the parameter matrix.

$\displaystyle\begin{array}[]{l}p_{w_{\textit{start}}}=\operatorname{Softmax}(% \mathbf{W}_{w_{\textit{start}}}\tanh W_{\textit{start}})\\ p_{w_{\textit{end}}}=\operatorname{Softmax}(\mathbf{W}_{w_{\textit{end}}}\tanh W% _{\textit{end}})\end{array}$ (15)

where $\mathbf{W}_{w_{\textit{start}}}$ and $\mathbf{W}_{w_{\textit{end}}}$ represent the parameter matrix.

W-no module predicts the number of columns in the where clause.

$\displaystyle p_{w_{no}}=\operatorname{Softmax}(\mathbf{W}_{w_{no}}\tanh W_{no})$ (16)

where $\mathbf{W}_{w_{no}}$ represents the parameter matrix.

3.5 Training and inference

In the course of the training process, our focus centers on optimizing the objective function, which comprises the amalgamation of losses associated with individual sub-tasks. Specifically, for the tasks denoted as S-col, S-agg, W-col, W-no, W-op, and W-val, we employ the cross-entropy loss. Notably, the framework outlined in this paper is rooted in the utilization of the pre-trained RoBERTa model. To facilitate the acquisition of a holistic feature representation by the HFD model prior to its decoupling stage, we directly input the output features generated by RoBERTa into the slot prediction module. This incorporation introduces a regular term via the application of the cross-entropy loss, thereby constraining the feature representation originating from RoBERTa.

It is worth acknowledging that during the inference process when converting text to SQL, the predicted SQL queries may exhibit errors related to schema linkage or syntax. To address these issues, we draw inspiration from the concept of execution-guided decoding, as introduced by Wang et al. [33] in their work on execution-guided decoding. The proposed methodology leverages iterative execution, involving a sequence of SQL queries from the candidate list being passed to the executor, with subsequent elimination of those queries that cannot be executed or return null results. In line with this approach, we incorporate execution-guided decoding subsequent to the model’s initial prediction.

4. Experiment

4.1 Dataset

In our study, we employ the WikiSQL dataset, a single-table-based text-to-SQL dataset meticulously documented by Zhong et al. under the reference [43]. This dataset comprises a substantial corpus of 56,355, 8,421, and 15,878 question-SQL query pairs allocated for the purposes of training, development, and testing, respectively. An essential characteristic of the dataset is the stringent division of tables between training and development sets, ensuring that any table found in either of these sets will not be replicated in the test set.

4.2 Evaluation metrics

Two prevailing evaluation metrics employed in the assessment of text-to-SQL systems are as follows: the first metric pertains to Logic Form Accuracy (LFA), which serves the purpose of ascertaining the fidelity of SQL statements engendered by the model with respect to their correctness. The quantification of LFA is facilitated by the mathematical formulation presented in Eq. (18).

$\displaystyle\text{Score }_{lf}=\left\{\begin{array}[]{ll}1&\textit{SQL}^{% \prime}=\textit{SQL}\\ 0&\textit{SQL}^{\prime}\neq\textit{SQL}\\ \end{array}\right.$ (17) $\displaystyle\textit{Acc}_{lf}=\frac{1}{N}\sum_{n=1}^{N}\textit{Score}_{lf}^{n}$ (18)

The subsequent objective involves ascertaining the concordance between the anticipated outcomes of SQL statement executions and the actual results achieved during SQL statement executions, with particular emphasis on assessing the Execution Accuracy, as elucidated by the mathematical expression presented in Eq. (20).

$\displaystyle\text{Score}_{ex}=\left\{\begin{array}[]{ll}1&Y^{\prime}=Y\\ 0&Y^{\prime}\neq Y\end{array}\right.$ (19) $\displaystyle\textit{Acc}_{ex}=\frac{1}{N}\sum_{n=1}^{N}\textit{Score}_{ex}^{n}$ (20)

Within the formula, denoted as $\textit{Acc}_{lf}$ , is the indicator for logical form accuracy, while $\textit{Acc}_{ex}$ represents the measure of execution accuracy. The variable $N$ signifies the quantity of data under consideration, whereas $\textit{SQL}^{\prime}$ and SQL respectively symbolize the anticipated SQL statement and the authentic SQL statement. Additionally, $Y^{\prime}$ and $Y$ stand for the outcomes of executing the anticipated SQL statement and the genuine SQL statement. It is important to note that the very same performance metrics shall be employed within this research to assess the efficacy of the proposed model.

4.3 Training details

The present study represents a notable advancement over the prior research endeavor, known as HydraNet, as delineated in the seminal work by Lyu et al. [17]. The investigation herein is underpinned by the instantiation of the PyTorch deep learning framework, featuring an intricately orchestrated training regimen spanning ten epochs. A learning rate of 3e-5 is employed, accompanied by a judiciously selected batch size of 128. The optimization process is orchestrated via the AdamW algorithm, with an additional safeguard against overfitting being realized through the deployment of the Dropout regularization method. Notably, all experimental facets transpire within the computational realm, orchestrated across a triad of Nvidia GeForce RTX 3090Ti Graphics Processing Units, thereby attesting to the computational intensity underpinning this inquiry.

4.4 Compared models

The current study presents a comparative analysis of the novel method under investigation in relation to several contemporary state-of-the-art models. These models include: (1) Seq2SQL, characterized as a generation-based approach in natural language interface to databases (NLIDB) [43]; (2) SQLnet, acknowledged as a sketch-based methodology for semantic parsing in NLIDB systems [37]; (3) TypeSQL, an innovative model that leverages type information for enhanced comprehension of infrequent entities and numerical values within user queries [41]; (4) Coarse-to-Fine, a sophisticated neural architecture, designed to address the semantic parsing task in two distinct stages, thereby enhancing structural awareness and interpretability [5]; (5) SQLova, a pioneering model that integrates pre-trained language models into the sketch-based approach for NLIDB [11]; (6) X-SQL, which enriches the representation of structural database schemas by incorporating contextual embeddings [10]; (7) HydraNet, an innovative approach that partitions the NLIDB task into column-wise ranking and decoding processes [17]; (8) IE-SQL, a distinctive model proposing an information extraction framework for Text-to-SQL conversion, emphasizing sequence-labeling-based relation extraction techniques [18]; (9) QA-SQL, which reformulates the NLIDB task as a question-answering problem, employing a unified machine reading comprehension (MRC) model for multi-slot prediction [38]; (10) MC-SQL, a unique approach tailored for zero-shot text-to-SQL conversion tasks, notable for its independence from additional manual annotations [3]; (11) RLC, a model that simplifies the decoder layer of existing methods and introduces a novel loss re-weighting technique for enhanced performance [36]; and (12) UniSAr, a recent extension of autoregressive language models, incorporating non-invasive structural enhancements to promote interpretability and performance in NLIDB tasks [6].

4.5 Experimental results

Table 2
Logical form (LF) and execution (EX) accuracy on WikiSQL dataset. The HFD’s batch size is set to 128 due to limitations in memory resources, and in the HydraNet model, we followed they set the batch size to 256. In their work, they describe that larger batch size gives better results.³ In the later section of the ablation experiments, we reproduced HydraNet at a batch size of 128 for comparison

	Dev		Test
Model	LF	EX	LF	EX
Seq2SQL	49.5	60.8	48.3	59.4
SQLNet	63.2	63.2	61.3	68.0
TypeSQL	68.0	74.5	66.7	73.5
Coarse-to-Fine	72.5	79.0	71.7	78.5
SQLova	81.6	87.2	80.7	86.2
X-SQL	83.8	89.5	83.3	88.7
QA-SQL	81.9	87.8	81.8	87.4
HydraNet₂₅₆	83.6	89.1	83.8	89.2
MC-SQL	84.1	89.7	83.7	89.4
RLC	84.3	90.4	83.9	89.7
UniSAr_T5-Large	85.9	91.2	8.1	91.9
UniSAr_BART-Large	86.7	91.7	85.8	91.4
HFD₁₂₈	84.4	89.6	83.9	89.0
SQLova_EG	84.2	90.2	83.6	89.6
X-SQL_EG	86.2	92.3	86.0	91.8
HydraNet_{EG_256}	86.6	92.4	86.5	92.2
HFD_{EG_128}	87.1	92.6	86.5	92.0

In this study, we conduct a comprehensive comparative analysis of our proposed methodology against established models that have previously reported their performance in the context of the WikiSQL task. A summary of this comparison is provided in Table 2. Specifically, our method is contrasted with the Sequence-to-sequence model Seq2SQL [37] and the syntax-based approach Coarse-to-Fine [5]. Notably, we also assess our approach against several sketch-based slot-filling techniques, including SQLNet [37], SQLova [11], TypeSQL [41], X-SQL [10], HydraNet [17], QA-SQL [38], MC-SQL [3], and RLC [36], which exhibit similarities to the HFD model proposed in this manuscript.

It is imperative to emphasize that the HFD model places a heightened emphasis on feature decoupling across subtasks, setting it apart from existing slot-filling-based methods. Our comparative analysis, as depicted in Table 2, reveals that our approach demonstrates a notably promising enhancement in the performance of logic form accuracy concerning SQL statements. Moreover, in the context of execution-guided decoding, our method exhibits superior performance when compared to existing approaches, including SQLova_EG, X-SQL_EG, and HydraNet_{EG_256}. Furthermore, we observe that our methodology holds its ground favorably when compared to methods employing larger pre-trained models, such as UniSAr_T5-Large and UniSAr_BART-Large.

4.6 Further analysis

4.6.1 Ablation experiments

Table 3
Logical form (LF) and execution (EX) accuracy on WikiSQL dataset in ablation experiments

	Dev		Test
Model	LF	EX	LF	EX
HFD	84.4	89.6	83.9	89.0
HFD-SFD	83.6	88.2	83.7	88.3
HFD-CFD-SFD	83.5	88.3	83.2	88.0
HydraNet₁₂₈	83.4	88.5	83.0	88.2
HFD_EG	87.1	92.6	86.5	92.0
HFD-SFD_EG	86.9	91.8	86.7	91.8
HFD-CFD-SFD_EG	86.4	91.5	86.2	91.3
HydraNet_EG_128	86.3	91.7	86.1	91.7

In this section, we present the findings of a series of ablation experiments, accompanied by a comprehensive description of the ablation model, which is expounded upon in the subsequent content.

•

HFD: It is the benchmark model, which contains all the components described in Section 3, which is our proposed hierarchical features decoupling model.

•

HFD-SFD: It removes the features decoupling of the slot-fillings in SELECT and WHERE clauses, and the part that’s removed is the approach we describe in Section 3.3.

•

HFD-SFD-CFD: It removes our proposed hierarchical features decoupling method, which merely ignores the irrelevance ranking score in HydraNet.

•

HydraNet₁₂₈: HFD is based on an improved version of HydraNet, so in the table, we show the difference between the two models for the same batch size.

Table 3 provides a comprehensive illustration of the noteworthy enhancements in Hierarchical Feature Decoupling (HFD) when contrasted with the benchmark model, HydraNet₁₂₈, which was chosen as the foundation for this study. Furthermore, the ablation experiments conducted herein substantiate the viability of our proposed methodology grounded in the concept of hierarchical feature decoupling. Significantly improved experimental outcomes are also observed with the deployment of a single-layer decoupling approach, HFD-SFD, in comparison to non-decoupled methods, specifically HFD-CFD-SFD and HydraNet₁₂₈. Our experimental findings underscore the essentiality of decoupling techniques among diverse subtasks within the domain of single-table-based text-to-SQL tasks. Furthermore, the conceptual framework outlined in this study offers a transferable paradigm for multi-task learning scenarios.

4.6.2 Slot type-wise results

Table 4
Fine-grained analysis for various methods on the test set of WikiSQL dataset

Model	$S_{\textit{col}}$	$S_{\textit{agg}}$	$W_{no}$	$W_{\textit{col}}$	$W_{op}$	$W_{\textit{val}}$
SQLova	96.8	90.6	98.5	94.3	97.3	95.4
X-SQL	97.2	91.1	98.6	95.4	97.6	96.6
QA-SQL	96.8	92.8	98.3	95.4	98.7	96.8
HydraNet₂₅₆	97.6	91.4	98.4	95.3	97.4	96.1
MC-SQL	96.4	90.6	98.8	97.8	97.8	96.9
HydraNet₁₂₈	97.6	91.3	98.2	94.9	97.2	95.6
HFD	97.6	91.5	98.0	95.1	97.3	96.0
HFD-SFD	97.7	91.4	97.9	95.0	97.1	95.9
HFD-CFD-SFD	97.6	91.2	98.0	94.9	97.2	95.8
SQLova_EG	96.5	90.4	97.0	95.5	95.8	95.9
X-SQL_EG	97.2	91.1	98.6	97.2	97.5	97.9
HydraNet_{EG_256}	97.6	91.4	98.4	97.2	97.5	97.6
HydraNet_{EG_128}	97.6	91.3	98.1	97.0	97.2	97.4
HFD_EG	97.6	91.5	98.0	97.0	97.3	97.4
HFD-SFD_EG	97.7	91.4	98.4	97.3	97.5	97.8
HFD-CFD-SFD_EG	97.6	91.2	98.2	97.1	97.4	97.4

To facilitate a comprehensive exploration of the operational efficacy of individual sub-modules within diverse computational methodologies, as exemplified in Table 4, an in-depth scrutiny was conducted. In this context, we designate $S_{\textit{col}}$ as the designated column identifier in the SELECT clause, while $S_{\textit{agg}}$ is indicative of the employed aggregation function. Concomitantly, $W_{no}$ represents the count of conditions present within the WHERE clause. Furthermore, the nomenclature $W_{\textit{col}}$ , $W_{op}$ , and $W_{\textit{val}}$ corresponds to the respective denotations for the column identifier, operator, and value entailed within a given conditional expression.

In comparison to HydraNet, HFD demonstrated results congruent with its performance in the slot prediction task. Specifically, concerning the HydraNet₂₅₆ variant, HFD exhibited a marked enhancement in its effectiveness across all subtasks. It is worth noting that resource limitations in terms of GPU capacity precluded the execution of experiments with a batch size of 256; nonetheless, it is evident from prior research that the model’s performance improves concomitantly with an increase in batch size.4

⁴

https://github.com/lyuqin/HydraNet-WikiSQL/issues/10.

Table 4 provides a comprehensive overview of the current experimental parameters, showcasing noteworthy accomplishments in terms of the aggregate (

S_{\textit{agg}}

) and column-wise (

S_{\textit{col}}

) scores. Ablation experiments further reveal that our method consistently outperforms the individual slot prediction tasks for each subtask, even when the utilization of execution-guided decoding is not taken into consideration.

5. Conclusion

This paper introduces an innovative multi-task framework denoted as Hierarchical Feature Decoupling (HFD), designed to address the intricacies of single-table-based text-to-SQL tasks. HFD incorporates a bifurcated feature decoupling strategy for subtasks related to the SELECT and WHERE clauses, wherein features specific to each subtask within these clauses are independently disentangled. The HFD model, as proposed, demonstrates noteworthy performance outcomes when evaluated on the WikiSQL benchmark.

Footnotes

Acknowledgments

The authors extend their gratitude to the anonymous reviewers for their valuable and perceptive commentary. Financial support for this research endeavor was provided by the National Natural Science Foundation of China (Grant No. 62176053). This research work is supported by the Big Data Computing Center of Southeast University.

References

Caruana

, Multitask learning, Machine Learning 28(1) (1997), 41–75.

Chang

Liu

Tang

Huang

Zhou

, Zero-shot text-to-sql learning with auxiliary task, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 2020, pp. 7488–7495.

Chen

Guo

Wang

Qiu

Wang

, Leveraging table content for zero-shot text-to-sql with meta-learning, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, 2021, pp. 3992–4000.

Chen

Badrinarayanan

Lee

C.-Y.

Rabinovich

, Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks, in: International Conference on Machine Learning, PMLR, 2018, pp. 794–803.

Dong

Lapata

, Coarse-to-fine decoding for neural semantic parsing, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 731–742.

Dou

Gao

Pan

Wang

Che

Lou

J.-G.

Zhan

, Unisar: A unified structure-aware autoregressive language model for text-to-sql semantic parsing, International Journal of Machine Learning and Cybernetics, 2023, 1–16.

Fang

Zhang

X.-Y.

Bai

et al., Dynamic multi-task learning with convolutional neural network, in: IJCAI, 2017, pp. 1668–1674.

Ghosh

P.K.

Dey

Sengupta

, Automatic sql query formation from natural language query, International Journal of Computer Applications 975 (2014), 8887.

Guo

Zhan

Gao

Xiao

Lou

J.-G.

Liu

Zhang

, Towards complex text-to-sql in cross-domain database with intermediate representation, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 4524–4535.

10.

Mao

Chakrabarti

Chen

, X-sql: reinforce schema representation with context, arXiv preprint arXiv:1908.08113, 2019.

11.

Hwang

Yim

Park

Seo

, A comprehensive exploration on wikisql with table-aware word contextualization, arXiv preprint arXiv:1902.01069, 2019.

12.

Kate

Kamble

Bodkhe

Joshi

, Conversion of natural language query to sql query, in: 2018 Second International Conference on Electronics, Communication and Aerospace Technology (ICECA), IEEE, 2018, pp. 488–491.

13.

Kenton

J.D.M.-W.C.

Toutanova

L.K.

, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of NAACL-HLT, 2019, pp. 4171–4186.

14.

Liu

Ran

Guo

et al., Decoupling makes weakly supervised local feature better, in: CVPR, 2022.

15.

Liu

Chen

Gao

, Multi-task deep neural networks for natural language understanding, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 4487–4496.

16.

Liu

Ott

Goyal

Joshi

Chen

Levy

Lewis

Zettlemoyer

Stoyanov

, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692, 2019.

17.

Lyu

Chakrabarti

Hathi

Kundu

Zhang

Chen

, Hybrid ranking network for text-to-sql, arXiv preprint arXiv:2008.04759, 2020.

18.

Yan

Pang

Zhang

Shen

, Mention extraction and linking for sql query generation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 6936–6942.

19.

Misra

Shrivastava

Gupta

Hebert

, Cross-stitch networks for multi-task learning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 3994–4003.

20.

Norouzifard

Davarpanah

Shenassa

et al., Using natural language processing in order to create sql queries, in: 2008 International Conference on Computer and Communication Engineering, IEEE, 2008, pp. 600–604.

21.

Chaudhari

P.P

, Natural language statement to sql query translator, International Journal of Computer Applications 82(5) (2013), 18–22.

22.

Qin

Liu

, Feature projection for improved text classification, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 8161–8171.

23.

Rao

Agarwal

Chaudhry

Kulkarni

Patil

, Natural language query processing using semantic grammar, International Journal on Computer Science and Engineering 2(2) (2010), 219–223.

24.

Reinaldha

Widagdo

T.E.

, Natural language interfaces to database (nlidb): Question handling and unit conversion, in: 2014 International Conference on Data and Software Engineering (ICODSE), IEEE, 2014, pp. 1–6.

25.

Shi

Tatwawadi

Chakrabarti

Mao

Polozov

Chen

, Incsql: Training incremental text-to-sql parsers with non-deterministic oracles, arXiv preprint arXiv:1809.05054, 2018.

26.

Singh

Solanki

, An algorithm to transform natural language into sql queries for relational databases, Selforganizology 3(3) (2016), 100–116.

27.

Stooke

Lee

Abbeel

Laskin

, Decoupling representation learning from reinforcement learning, in: International Conference on Machine Learning, PMLR, 2021, pp. 9870–9879.

28.

Tang

Liu

Zhao

Gong

, Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations, in: Fourteenth ACM Conference on Recommender Systems, 2020, pp. 269–278.

29.

Thung

K.-H.

Wee

C.-Y.

, A brief review on multi-task learning, Multimedia Tools and Applications 77(22) (2018), 29705–29725.

30.

Utama

Weir

Basik

Binnig

Çetintemel

Hättasch

Ilkhechi

Ramaswamy

Usta

, An end-to-end neural natural language interface for databases, arXiv preprint arXiv:1804.00401, 2018.

31.

Vaswani

Shazeer

Parmar

Uszkoreit

Jones

Gomez

A.N.

Kaiser

Ł.

Polosukhin

, Attention is all you need, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, pp. 6000–6010.

32.

Wang

Shin

Liu

Polozov

Richardson

, Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7567–7578.

33.

Wang

Huang

P.-S.

Polozov

Brockschmidt

Singh

, Execution-guided neural program decoding, in: ICML Neural Abstract Machines & Program Induction Workshop, 2018, 2018.

34.

Wang

Zhuang

Gao

Wang

Zou

, Learning decoupling features through orthogonality regularization, in: ICASSP 2022 – 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 7562–7566.

35.

Wang

Zhang

Guo

Yin

Chen

, Decoupling representation learning and classification for gnn-based anomaly detection, in: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021, pp. 1239–1248.

36.

Wei

Huang

, Enhance text-to-sql model performance with information sharing and reweight loss, Multimedia Tools and Applications 81(11) (2022), 15205–15217.

37.

Liu

Song

, Sqlnet: Generating structured queries from natural language without reinforcement learning, arXiv preprint arXiv:1711.04436, 2017.

38.

Yan

Zhang

Shen

, Sql generation via machine reading comprehension, in: Proceedings of the 28th International Conference on Computational Linguistics, 2020, pp. 350–356.

39.

Yang

Hospedales

, Deep multi-task representation learning: A tensor factorisation approach, in: 5th International Conference on Learning Representations, 2017.

40.

Yang

Dai

Yang

Carbonell

Salakhutdinov

R.R.

Q.V.

, Xlnet: Generalized autoregressive pretraining for language understanding, Advances in Neural Information Processing Systems 32 (2019), 5753–5763.

41.

Zhang

Radev

, Typesql: Knowledge-based type-aware neural text-to-sql generation, in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 2018, pp. 588–594.

42.

Yasunaga

Yang

Zhang

Wang

Radev

, Syntaxsqlnet: Syntax tree networks for complex and cross-domain text-to-sql task, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 1653–1663.

43.

Zhong

Xiong

Socher

, Seq2sql: Generating structured queries from natural language using reinforcement learning, arXiv preprint arXiv:1709.00103, 2017.

HFD: Hierarchical feature decoupling for SQL generation from text

Abstract

Keywords

1. Introduction

Table 1 Contents in database table 1-10416547-1

2.1 Natural language to structured query language

2.2 Multi-task learning

3. Hierarchical feature decoupling model

4. Experiment

4.1 Dataset

4.2 Evaluation metrics

4.4 Compared models

4.5 Experimental results

4.6.1 Ablation experiments

Table 3 Logical form (LF) and execution (EX) accuracy on WikiSQL dataset in ablation experiments

Table 4 Fine-grained analysis for various methods on the test set of WikiSQL dataset

Footnotes

Acknowledgments

References

Table 1
Contents in database table 1-10416547-1

Table 3
Logical form (LF) and execution (EX) accuracy on WikiSQL dataset in ablation experiments

Table 4
Fine-grained analysis for various methods on the test set of WikiSQL dataset