Sampleformer: An efficient conformer-based Neural Network for Automatic Speech Recognition

Abstract

The Convolution-augmented Transformer (Conformer) model, which was recently introduced, has attained state-of-the-art(SOTA) results in Automatic Speech Recognition (ASR). In this paper, a series of methodical investigations uncover that the Conformer’s design decisions may not represent the most efficient choices when operating within the constraints of a limited computational budget. After a thorough re-evaluation of the Conformer architecture’s design choices, we propose Sampleformer which reduces the Conformer architecture complexity and has more robust performance. We introduce downsampling to the Conformer Encoder, and to exploit the information in the speech features, we incorporate an additional downsampling module to enhance the efficiency and accuracy of our model. Additionally, we propose a novel and adaptable attention mechanism called multi-group attention, effectively reducing the attention complexity from $O(n^{2}d)$ to $O(n^{2}d\cdot f/g)$ . We performed experiments on the AISHELL-1 corpora, our 13.3 million-parameter CTC model demonstrates a 3.0%/2.6% relative reduction in character error rate (CER) on the dev/test sets, all without the utilization of a language model (LM). Additionally, the model exhibits a 30% improvement in inference compared to our CTC Conformer baseline and trains 27% faster.

Keywords

Speech recognition conformer attention mechanism complexity reduction

1. Introduction

End-to-end neural network models have achieved substantial advancements in numerous automatic speech recognition tasks and have become the established standard for SOTA approaches. Convolutional neural networks (CNN) [1, 2, 3, 4] and Transformers [5, 6, 7, 8, 9] have several limitations as popular backbone architectures for ASR models. Generally, Transformers involve prohibitive computing and memory overhead, whereas CNN models have limited ability to effectively capture global contexts. To overcome these shortcomings, existing techniques have proposed different approaches to address these problems, such as pruning [10], quantization [11], optimized architecture design [12], and knowledge distillation [13]. These methods can reduce the complexity of the model, whether used individually or together. Besides, the availability of extensive, manually annotated datasets has enabled the training of robust deep neural networks for ASR. However, a significant drawback of employing a powerful deep neural network in practical applications is the associated resource cost. Achieving high performance demands a substantial training budget and the utilization of a considerable number of GPUs [1, 2].

Recently, Conformer [14] has proposed a novel convolution-augmented Transformer architecture, which combined CNN and self-attention. Due to its capability to simultaneously capture both global and local dependencies within audio signals, the Conformer architecture has achieved SOTA results for various end-to-end ASR tasks [15, 16]. Certainly, despite its significance as a foundational architecture in ASR tasks, the Conformer still exhibits certain limitations that warrant further improvement. First, the efficiency of Conformer on long sequence lengths is still limited by the quadratic complexity of the attention mechanism [17]. Furthermore, the Conformer architecture integrates the Macaron structure [14, 18], activation functions, and multiple different normalization schemes, which is more complex compared to the conventional Transformer structure employed in other domains [8, 9, 19]. Related research has been proposed in order to address the difficulty of efficiently deploying the model and reasoning on the dedicated hardware platforms [20, 21]. [22] demonstrates the effective incorporation of progressive downsampling into convolution-augmented transformer networks. [23] simplifies the Conformer into a more straightforward block structure reminiscent of standard Transformer blocks [8, 9] by employing a temporal U-Net structure to reduce computational complexity. Adopting a design such as sampling to reduce computation is an effective approach. More importantly, this raises the question of whether it is necessary and optimal to adopt such design choices to achieve good performance for ASR tasks.

In this paper, we proposed a more efficient hybrid convolution-attention architecture named Sampleformer. It has been found that temporal redundancy increases with network depth. To address this issue, a novel downsampling layer has been introduced to reduce redundancy between neighbouring features. Additionally, an SE module has been added to the convolutional downsampling module to selectively enhance informative features. we propose a novel and flexible attention mechanism called multi-group attention to solving the computational asymmetry caused by the introduction of downsampling. Our primary objective is to design an efficient architecture aimed at reducing the complexity of the Conformer, achieving a lower CER within a specific computational budget. In particular, the following contributions are made to our proposed Sampleformer model:

We find that the learned feature representations of temporally close speech frames correspond to very similar internal representations in Conformer training, especially deeper in the network. To address this, we proposed a novel structure inspired by the structure of Efficient Conformer [22]. In this structure, two downsampling layers halve the sampling rate at the network’s midpoint, and the Conformer block structure between the two downsampling layers adopted a residual connection.

We redesign the downsampling architecture based on our observation that convolutional downsampling is unable to effectively model the local and global dependencies and also results in performance degradation for ASR tasks. We designed a novel downsampling layer that combines the two designs of squeeze-and-excitation networks (SE-Net) [24] and hybrid attention-convolution architecture [14]. This design aims to enhance accuracy, even though it incurs some additional computational resources.

We finely examine the architecture of the network. The complexity of attention increases quadratically as the sequence length grows, we find that adopting downsampling methods introduces computation asymmetry into attention-based architectures and lead to a bottleneck in processing time. One solution involves creating an efficient self-attention mechanism [25, 26, 27], whereas another alternative to conventional attention is adopting local attention [28, 29]. We draw inspiration from these methods and introduce a novel and adaptable attention mechanism, which we term “multi-group attention”. Along the feature dimension, multi-group attention repeatedly groups adjacent time elements within the sequence, the grouping results in a reduction of attention complexity from $O(n^{2}d)$ to $O(n^{2}d\cdot f/g)$ , where $n$ represents the sequence length, $d$ stands for the hidden dimension, and $f$ and $g$ are the group size parameters.

We substantiate the final model architecture by conducting an ablation study, aiming to gain a deeper understanding of the enhancements resulting from the various methods.

Figure 1.

(Left) The Conformer architecture and (Right) the Sampleformer architecture which comprises several Conformer blocks using multi-group attention, the hybrid convolution-attention-style block structure. The encoded sequence undergoes multiple downsampling operations before being projected into wider feature dimensions.

2. Previous work

Typically, modern end-to-end ASR models consist of an encoder and a decoder. The encoder’s architecture plays a pivotal role in defining the representational capacity of an ASR model and its capacity to extract acoustic features, which is crucial for the overall performance of the ASR model. CNN is widely favored as the foundational model architecture for addressing the time-frequency characteristics of speech. In hybrid CTC-Attention neural architecture [30], the VGG network architecture, consisting of a 4-layer CNN and a 2-layer MaxPool, remains a fundamental component of the encoder model. Deep CNN models were first explored in end-to-end neural systems [4, 31], and further improved by introducing SE-Net in ContextNet [3] and depth-wise separable convolution (DSC) [32, 33] in QuartzNet [1]. However, CNN often fails to capture the global context.

A novel model architecture named Conformer [14], has achieved good performance in ASR tasks by effectively modeling both global and local dependencies through the convolutional-augmented transformer. However, the attention layers require way more computation for longer sequence lengths. The attention layer’s quadratic complexity remains excessively high. Similar approaches have been proposed to decrease the computational demands of multi-head attention (MHA) in ASR [17, 34, 35, 36]. Efficient Conformer [22] introduces a progressive downsampling scheme to minimize the training cost for the Conformer model. Additionally, it incorporates grouped attention to reduce inference time. Squeezeformer [23] introduces a scheme similar to progressive downsampling but also introduces an upsampling mechanism. This design concept takes inspiration from both the U-Time [37] architecture used in sleep signal analysis and the U-Net [38] architecture in computer vision. Nextformer [39] incorporates an extra downsampling module aimed at extracting fine-grained time-frequency speech features, and the module also serves to replace the subsampling component within a conformer-based attention-based encoder-decoder (AED). However, downsampling methods can reduce the training and inference costs of the model. It is important to note that temporal downsampling may lead to training instability and divergence, which can weaken the model’s representational capabilities. Our work incorporates similar progressive downsampling, and redesigning novel downsampling layers that allow us to achieve better performance and faster decoding. Our primary objective is to design an efficient architecture aimed at reducing the complexity of the Conformer. Moreover, we propose a novel and flexible attention mechanism called multi-group attention to solving the computational asymmetry caused by the introduction of downsampling.

3. Methods

The speech community has commonly adopted the Conformer architecture as a foundational framework for a variety of ASR tasks. At the macro-level, the Conformer is constructed by stacking multiple blocks of a Macaron structure [14, 18], with each structure comprising four modules per block, as shown in Fig. 1 (left). In this paper, we systematically reconsider the design choices within the Conformer by incorporating downsampling and designing novel downsampling layers. These enhancements enable us to attain better performance and faster decoding. Moreover, in order to enhance the efficiency of the early self-attention layers, multi-group attention is employed to strike a balance between the model’s overall complexity and accuracy, ensuring no compromise in precision. Conformer-CTC-S has been selected as the baseline model for our case study, and we have compared the CER for each architecture using the Aishell-1 dataset [40] as a performance metric.

3.1 Macro-architecture design

Although empowered by the hybrid attention-convolution structure to capture both local and global dependencies, it’s important to note that the attention operation demonstrates quadratic complexity concerning the input sequence length. Within the Conformer model, a decrease in the input sampling rate from 10 ms to 40 ms is achieved through convolutional subsampling blocks, while maintaining a constant temporal scale for all attention and convolution operations. Inspired by recent work on ASR, [2, 3, 22] reduce CNN computation for faster training and inference by applying progressive subsampling, we experimentally introduce downsampling into the Conformer encoder, and the proposed Sampleformer encoder is shown in Fig. 1 (right). In particular, [23] reveals that temporal redundancy increases as the network depth. We conduct this analysis using the Conformer model on feature embeddings for each speech frame. Fifty audio signals are randomly sampled from Aishell-1’s dev dataset, and recording activations of the Conformer blocks. We calculate the average cosine similarity between adjacent embedding vectors, with the results depicted in Fig. 5. We observe that at the topmost network layer (Layer 16), the cosine similarity between embedding vectors for adjacent speech frames at a distance of 1 or 2 exceeds 90%, and even the similarity between four speech frames is more than 65%. This indicates a growing temporal redundancy with deeper networks, especially for architectures that are built by stacking Conformer blocks multiple times. We hypothesize that the elimination of redundancies in feature embedding vectors can reduce unnecessary computational overhead without compromising accuracy.

In our macro-architecture improvement step, audio features are initially downsampled using convolutional downsampling and then fed into the encoder, consisting of multiple Conformer blocks with the same feature dimension. We keep the sampling rate to be 20 ms up to the 5th block, then reduce the sampling rate of each input sequence to 40 ms by replacing the original convolutional module with a convolutional downsampling module, and then reduce the sampling rate of each input sequence to 80 ms at the 10th block. Similar to spatial downsampling in computer vision models, this temporal downsampling technique is commonly utilized to economize computational resources and generate hierarchical-level features [41, 42, 43, 44].

However, temporal downsampling results in unstable and divergent training behavior. One possible reason is that after downsampling the rate to 80 ms, the decoder lacks sufficient resolution to successfully decode the full sequence. Taking inspiration from successful computer vision architectures like ResNet [42], U-Net [38], and U-Time [37], we modified the model’s architecture, as illustrated in Fig. 1 (right). This allows the model to train stably and gives better results, which can be found in our experiments.

Figure 2.

Convolution downsampling module. The residual module contains a 1D scSE-Net and the pointwise projection layer, where the pointwise projection projects the input and output to the same dimension. Sequence downsampling is achieved through the application of a strided depthwise convolution [22].

3.2 Micro-architecture design

Up to this point, we have established Sampleformer’s macro-structure. In this subsection, our attention shifts to optimizing the micro-structure of each module to enhance efficiency and performance.

3.2.1 Convolutional downsampling module

We experimentally introduce downsampling into Conformer Encode by replacing the original convolutional module with a convolutional downsampling module illustrated in Fig. 2. By incorporating a novel architectural unit, namely the “squeeze-and-excite” (SE) module [24, 45], we depict the structure in Fig. 3. scSE blocks can simultaneously recalibrate the input both channel-wise and spatially, acquiring the ability to leverage global information to selectively enhance informative features while suppressing less valuable ones. Mathematically, for an input $x$ to a scSE block, the resulting output $y$ of the block can be formulated as:

$\displaystyle\textit{scSE}(x)=\theta_{1}(x)\circ x_{C}+\theta_{2}(x)\circ x_{T}$ (1)

where,

$\displaystyle\begin{aligned} &\displaystyle\theta_{1}(x)=\textit{Sigmoid}(\bm{% W_{1}}x_{c}+b_{1})\\ &\displaystyle\theta_{2}(x)=\textit{Sigmoid}(\bm{W_{3}}(Act(\bm{W_{2}}\bar{x}_% {t}+b_{2}))+b_{3})\\ &\displaystyle\bar{x}=\frac{1}{T}\sum_{t}x_{t}\end{aligned}$ (2)

In Eq. (1), where $\circ$ represents element-wise multiplication, $\theta_{1}(x)$ is the spatial information importance weight, $\theta_{2}(x)$ is a global channel-wise weight. In Eq. (2), $\textit{Sigmoid}(\cdot)$ denotes the sigmoid function, $\textit{Act}(\cdot)$ represents the activation function, $\bm{W_{1}}$ , $\bm{W_{2}}$ , $\bm{W_{3}}$ are weight matrics, and $b_{1}$ , $b_{2}$ , $b_{3}$ are bias vectors.

SE networks can be created by merely stacking a set of SE blocks or by substituting them for the original blocks at any depth in the architecture. The advantages of feature recalibration from SE blocks can accumulate across the entire network. We add an SE block on the convolutional downsampling module, as shown in Fig. 2. The deeper Conformer block selectively learns informative features, and the redundancy in the feature embedding can be further reduced without loss of accuracy. As shown in Fig. 5, Sampleformer with the addition of the scSE block can reduce the redundancy in the feature embedding, and the accuracy can also be improved, which can be found in our experiments.

Figure 3.

1D scSE module. This model comprises a fusion of both the Spatial Squeeze and Channel Excitation Block (cSE) and the Channel Squeeze and Spatial Excitation Block (sSE) [24, 45], which concurrently recalibrate the input channel-wise and spatially, we apply the concept to the 1D case.

3.2.2 Multi-group attention

In general, downsampling is achieved through the use of strided convolution or pooling operations. In Efficient Conformer [22], Multi-head self-attention (MHSA) is employed as a global downsampling operation, accomplished by substituting the standard MHSA layers with strided MHSA layers within each downsampling block. By employing relative sinusoidal position encoding, the self-attention module improves generalization across different input lengths. For relative multi-head self-attention, attention mechanism is conducted separately for multiple heads $H$ to a hidden dimension $X\in\mathbb{R}^{n\times d}$ as follows:

$\displaystyle\begin{aligned} &\displaystyle\textit{RelativeMHSA}(X)=\textit{% Concat}(O_{1},\ldots,O_{H})\bm{W^{O}}\\ &\displaystyle O_{h}=\textit{softmax}\left(\frac{\bm{Q_{h}K_{h}^{T}}+S_{h}^{% \textit{rel}}}{\sqrt{d_{h}}}\right)\bm{V_{h}}\end{aligned}$ (3)

Equation (3) is the encoder output after the MHSA Module. where $O_{h}=X\bm{W_{h}^{Q}}$ , $K_{h}=X\bm{W_{h}^{K}}$ and $V_{h}=X\bm{W_{h}^{V}}$ are query, key and value linear projections with parameter matrices $\bm{W_{h}^{Q}}$ , $\bm{W_{h}^{K}}$ , $\bm{W_{h}^{V}}\in\mathbb{R}^{d\times d_{h}}$ and $\bm{W^{O}}\in\mathbb{R}^{d\times d}$ represents the output linear projection matrix. $S^{\textit{rel}}\in\mathbb{R}^{n\times n}$ corresponds to a matrix of relative position scores.

Figure 4.

Multi-group attention. Multi-group attention reduces attention complexity from $O(n^{2}d)$ to $O(n^{2}d\cdot f/g)$ through the repeated grouping of adjacent temporal elements along the feature dimension. In this context, $n$ represents the sequence length, $d$ stands for the hidden dimension, and $f$ and $g$ denote the group size parameters. Multi-group attention is equivalent to standard attention when $g=f$ .

Yet, the application of the downsampling method to an attention-based architecture brings about the primary drawback of introducing computational asymmetry into the network. By adjusting the feature dimensions of the blocks, it becomes possible to achieve various downsampling stages with similar per-hidden-layer complexity for different network depths. However, the time bottleneck still exists. We propose to address this issue through the introduction of a novel and adaptable attention mechanism, referred to as multi-group attention, as depicted in Fig. 4. Multi-group attention reduces attention complexity from $O(n^{2}d)$ to $O(n^{2}d\cdot f/g)$ through the repeated grouping of adjacent temporal elements along the feature dimension, where $n$ represents the sequence length, $d$ stands for the hidden dimension, and $f$ and $g$ are the group size parameters. Attention queries, values, keys are reshaped from $\bm{Q}$ , $\bm{V}$ , $\bm{K}\in\mathbb{R}^{n\times d}$ to $\bm{Q}^{\textit{mulgrp}}$ , $\bm{V}^{\textit{mulgrp}}$ , $\bm{K}^{\textit{mulgrp}}\in\mathbb{R}^{n^{\prime}\times d^{\prime}}$ where $n$ transforms into $n^{\prime}=n\cdot f/g$ and $d$ becomes $d^{\prime}=d\cdot g/f$ . The output for a head $h$ is as follows:

$\displaystyle O_{h}^{\textit{mulgrp}}=\textit{softmax}\left(\frac{\bm{Q_{h}}^{% \textit{mulgrp}}\bm{K_{h}}^{\textit{mulgrpT}}+S_{h}^{\textit{rel}}}{\sqrt{d^{% \prime}_{h}}}\right)\bm{V_{h}}^{\textit{mulgrp}}$ (4)

Equation (4) is the encoder output after the multi-group attention. And concatenated multi-group attention output $O^{\textit{mulgrp}}\in\mathbb{R}^{n^{\prime}\times d^{\prime}}$ is reshaped to $O\in\mathbb{R}^{n\times d}$ prior to reaching the output projection layer. Multi-group attention is motivated by neighboring elements encoding similar features. Beginning with the earlier attention layers, where the encoded sequence is at its longest, we employ Multi-Group multi-head self-attention. This demonstrates our capability to mitigate computation asymmetry while maintaining accuracy.

4. Experiments

4.1 Data and training setup

Data

In this paper, we conduct model training and evaluation on the publicly available AISHELL-1 corpus [40] for Mandarin speech recognition. The AISHELL-1 consists of approximately 178 hours of labeled speech recordings from 400 speakers captured using high-fidelity microphones. Speakers come from diverse accents in China and the speech content is related to intelligent households, industrial production, drones, and other areas. For data augmentation, we apply SpecAugment [46, 47] with specific parameters to prevent overfitting during training, including frequency mask size parameters ( $F=10$ ) and the count of time and frequency masks ( $mT=mF=2$ ). Additionally, we create a byte-pair encoding tokenizer using SentencePiece [48] based on AISHELL-1 transcripts. The input features consist of 80-dimensional mel-scale log filter bank spectrograms computed using a 25 ms window and a 10 ms shift.

Training setup

Following the procedure detailed in Section 3, we construct Sampleformer variants of varying sizes. Specifically, we apply the corresponding macro and micro-architecture adjustments to create Sampleformer-S and M from Conformer-S and M, maintaining the model size. We conduct experiments using CTC models with 14M and 33M parameters, and Table 1 provides an overview of our model’s hyper-parameters.

Table 1
Detailed architecture configurations for Sampleformer and Conformer-CTC (baseline)

Model	Encoder blocks	Encoder dims	Attention heads	Conv kernel size	Group size ( $g$ , $f$ )	Params (M)
Conformer-CTC-S	16	176	4	31	–	13.0
Sampleformer-S	15	120, 168, 240	4	15	4, 2	13.3
Conformer-CTC-M	18	256	4	31	–	30.6
Sampleformer-M	16	180, 256, 360	4	15	4, 2	33.4

We train CTC models on the AISHELL-1 dataset for 400 epochs, utilizing a batch size of 32. This training is executed on a single Nvidia RTX 3090 GPU. We utilize the Adam Optimizer [49] and implement a transformer learning rate schedule [9], featuring $10K$ warm-up steps and a peak learning rate of $0.02/\sqrt{d_{\textit{enc}}}$ for the CTC task, where $d_{\textit{enc}}$ is the encoder output dimension. We use the gradient accumulation [50] during training, where the gradients are updated every 2 batches. We utilize a $P_{\textit{drop}}=0.1$ rate while applying $L2$ regularization with a weight of $1e^{-6}$ to model’s trainable weights. This adjustment enhances the model’s generalizability and mitigates the risk of overfitting during training.

The baseline dataloader sorts and packs utterances based on frame length and subsequently randomly selects them to pass to the model, optimizing training efficiency and maximizing GPU memory utilization. In many previous works [5, 51, 52, 53], the decoders of these models are frequently enhanced with an external language model (LM) for re-scoring the outputs to improve the final WER. A point of consideration during model evaluation lies in the incorporation of external LMs. Therefore, we conduct a fair comparison of the model architectures’ authentic representation capabilities and outcomes without external LMs, with performance metrics for the ASR models derived from our own replication of the best results.

4.2 Results of Aishell-1

Table 2 provides a CER comparison between Sampleformer, Conformer-CTC, and other SOTA CTC-based ASR models. The results suggest improvements in Sampleformer over previously published systems. Our Sampleformer-S model achieves competitive results of 5.46/5.71 without an LM for only 13.3M parameters. Furthermore, we achieved similar results to the version without multi-group attention, while realizing a 27% reduction in training time by employing multi-group attention with parameters $g=4$ , $f=2$ before the first downsampling.

Table 2
Comparison of AISHELL-1 CER (%) with recently published CTC models. At 13.3M parameters, Sampleformer-S outperforms the baseline model Conformer-S by 3.0%/2.6% on the dev/test dataset of the AISHELL-1. At 33.4M model parameters, the model still has nice results. The performance metrics for all models are derived from our dedicated efforts to reproduce the best achievable results

Model architecture	CER (%)		Params (M)
	Dev	Test
Conformer-CTC-S [14]	5.62	5.95	13.0
Squeezeformer-XS [23]	5.81	6.09	9.1
Eff. Conformer-CTC-S [22]	5.67	6.01	13.2
Sampleformer-S (ours)	5.63	5.89	13.3
w/o mutli-group Att	5.46	5.71	13.3
Conformer-CTC-M [14]	5.40	5.67	30.5
Squeezeformer-SM [23]	5.48	5.82	28.3
Eff. Conformer-CTC-M [22]	5.42	5.76	31.5
Sampleformer-M (ours)	5.41	5.66	33.4
w/o mutli-group Att	5.31	5.57	33.4

Figure 5.

Cosine similarity of embedding vectors between adjacent speech frames in the Conformer blocks. Growing temporal redundancy with deeper networks. In the Sampleformer structure, downsampling is applied to the temporal dimension following the 5th block and subsequently after the 10th block.

Table 3

Ablation studies for the design choices made in Sampleformer, including macro-architecture, Convolutional downsampling module, pointwise projection, and multi-group attention

Ablation	Model	CER (%)		Train time (h)
		Dev	Test
Ours	Sampleformer-S	5.63	5.89	90
Macro-Architecture	No downsampling block	5.85	6.16	107
	No skip connection	5.71	5.96	–
Convolutional downsampling module	No scSE block	5.84	6.46	–
	cSE block	5.68	6.13	–
	sSE block	5.73	6.21	–
Pointwise projection	Conv downsampling (Ours)	5.63	5.89	–
	AvgPool 1D	5.70	5.94	–
Attention	w/o mutli-group Att	5.46	5.71	122

Our Sampleformer model achieves satisfying results, with the CTC model converging faster with fewer epochs, achieving lower greedy CER. However, it still falls short of the original work, which benefited from training with larger batch sizes and more extensive resources.

4.3 Ablation studies

In this section, we will provide supplementary ablation studies analyzing the design decisions related to individual architectural components. We will utilize Sampleformer-S as the baseline model to better understand the improvements resulting from different approaches to composing the Sampleformer.

We first investigate the effects of employing downsampling. The downsampled architecture attains improved accuracy in CTC experiments, along with reduced multiply-add operations and shorter training time, as evidenced in the second major row of Table 3. Moreover, without skip connections, the performance of the model deteriorates. In the third major row of Table 3, Using the scSE block in the convolution module makes the accuracy improved. We experimented with two downsampling methods in the pointwise projection layer, using convolutional downsampling or Pooling downsampling, and we found that convolutional downsampling performs better than Pooling downsampling. Adapted to the network, the results show that the SE block allows for better application of MHSA to global downsampling operations.

Table 4
Ablation study on group size parameters

Group sizes ( $g$ , $f$ )	CER (%)		Train time (h)
	Dev	Test
(1, 1), (1, 1), (1, 1)	5.46	5.71	122
(4, 2), (1, 1), (1, 1)	5.63	5.89	90
(6, 2), (4, 2), (1, 1)	5.68	6.07	81
(12, 3), (6, 2), (4, 2)	5.74	6.33	72

Figure 6.

CTC models inference time for processing long sequences. The Sampleformer model shows an increase in inference speed with longer input sequences compared to the CTC Conformer baseline.

We incrementally expand the attention group size to analyze how multi-group attention impacts model complexity and recognition performance, as evidenced by the results in Table 4, highlighting its effectiveness in reducing model complexity without compromising recognition performance. By introducing multi-group attention in the early layers, the training speed is improved by 27% with average inference time accelerated by 30%, as shown in Fig. 6. By introducing multi-group attention with different attention group sizes at different depths of the network layers, the training speed can be further improved to 41%.

5. Conclusion

In this work, we have systematically examined the Conformer architecture and introduced a series of techniques to reduce the complexity of the Conformer and always obtain better performance within a certain computational budget. We introduce a new downsampling layer to reduce the temporal redundancy between adjacent features and to obtain better recognition performance with reduced computation. Moreover, we propose a novel and flexible attention mechanism called multi-group attention to solving the computational asymmetry caused by the introduction of downsampling. Finally, we establish the efficacy of our approach through an ablation analysis conducted on the Aishell-1 dataset. Trained on a limited computational budget of a single GPU, our 13.3M parameters Sampleformer model brings a 3.0%/2.6% improvement over the baseline model in dev/test, while achieving a 30% faster inference speed and a 27% shorter training time compared to our CTC Conformer baseline.

In the future, we intend to implement additional complementary complexity reduction methods, including distillation, quantization, and weight pruning to further decrease computational requirements.

Footnotes

Acknowledgments

This research was funded by the Science and Technology Key Projects of Guangxi Province, grant number 2020AA21077007; the Innovation Project of Guangxi Graduate Education, grant number YCSW2023019; the Guangxi New Engineering Research and Practice Project, grant number XGK2022003. The authors would like to express appreciation to Zhaohui Bu from the School of Foreign Languages of Guangxi University for her valuable comments and suggestions.

References

Kriman

et al., Quartznet: Deep automatic speech recognition with 1d time-channel separable convolutions, in: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2020, pp. 6124–6128.

Majumdar

et al., Citrinet: Closing the gap between non-autoregressive and autoregressive end-to-end models for automatic speech recognition, arXiv preprint arXiv:2104.01721, 2021.

Han

et al., Contextnet: Improving convolutional neural networks for automatic speech recognition with global context, arXiv preprint arXiv:2005.03191, 2020.

et al., Jasper: An end-to-end convolutional neural acoustic model, arXiv preprint arXiv:1904.03288, 2019.

Karita

et al., A comparative study on transformer vs rnn in speech applications, in: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2019, pp. 449–456.

Likhomanenko

et al., Rethinking evaluation in asr: Are our models robust enough, arXiv preprint arXiv:2010.11745, 2020.

Liu

et al., Improving RNN transducer based ASR with auxiliary tasks, in: 2021 IEEE Spoken Language Technology Workshop (SLT), IEEE, 2021, pp. 172–179.

Devlin

et al., Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018.

Vaswani

et al., Attention is all you need, Advances in Neural Information Processing Systems 30 (2017).

10.

Cantú-Paz

, Pruning neural networks with distribution estimation algorithms, in: Genetic and Evolutionary Computation – GECCO 2003: Genetic and Evolutionary Computation Conference Chicago, IL, USA, July 12–16, 2003 Proceedings, Part I, Springer, 2003, pp. 790–800.

11.

Dally

, High-performance hardware for machine learning, Nips Tutorial 2 (2015), 3.

12.

Tan

and Le

, Efficientnet: Rethinking model scaling for convolutional neural networks, in: International Conference on Machine Learning, PMLR, 2019, pp. 6105–6114.

13.

Buciluǎ

Caruana

and Niculescu-Mizil

, Model compression, in: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2006, pp. 535–541.

14.

Gulati

et al., Conformer: Convolution-augmented transformer for speech recognition, arXiv preprint arXiv:2005.08100, 2020.

15.

E.G.

et al., Pushing the limits of non-autoregressive speech recognition, arXiv preprint arXiv:2104.03416, 2021.

16.

Zhang

et al., Pushing the limits of semi-supervised learning for automatic speech recognition, arXiv preprint arXiv:2010.10504, 2020.

17.

Shim

Choi

and Sung

, Understanding the role of self attention for efficient speech recognition, in: International Conference on Learning Representations, 2022.

18.

et al., Understanding and improving transformer from a multi-particle dynamic system point of view, arXiv preprint arXiv:1906.02762, 2019.

19.

Radford

et al., Improving language understanding by generative pre-training, 2018.

20.

Kim

et al., I-bert: Integer-only bert quantization, in: International Conference on Machine Learning, PMLR, 2021, pp. 5506–5518.

21.

et al., Nn-lut: neural approximation of non-linear operations for efficient transformer inference, in: Proceedings of the 59th ACM/IEEE Design Automation Conference, 2022, pp. 577–582.

22.

Burchi

and Vielzeuf

, Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition, in: 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2021, pp. 8–15.

23.

Kim

et al., Squeezeformer: An efficient transformer for automatic speech recognition, arXiv preprint arXiv:2206.00888, 2022.

24.

Shen

and Sun

, Squeeze-and-excitation networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–7141.

25.

and Zhang

X.-L.

, Efficient conformer-based speech recognition with linear attention, in: 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), IEEE, 2021, pp. 448–453.

26.

Shen

et al., Efficient attention: Attention with linear complexities, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 3531–3539.

27.

Wang

Sun

Xie

and Ma

, Efficient conformer with prob-sparse attention mechanism for end-to-end speech recognition, arXiv preprint arXiv:2106.09236, 2021.

28.

Ramachandran

et al., Stand-alone self-attention in vision models, Advances in Neural Information Processing Systems 32 (2019).

29.

Parmar

et al, Image transformer, in: International Conference on Machine Learning, PMLR, 2018, pp. 4055–4064.

30.

Hori

Watanabe

Zhang

and Chan

, Advances in joint CTC-attention based end-to-end speech recognition with a deep CNN encoder and RNN-LM, arXiv preprint arXiv:1706.02737, 2017.

31.

Zhang

et al., Towards end-to-end speech recognition with deep convolutional neural networks, arXiv preprint arXiv:1701.02720, 2017.

32.

Howard

A.G.

et al., Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint arXiv:1704.04861, 2017.

33.

Sifre

and Mallat

, Rigid-motion scattering for texture classification, arXiv preprint arXiv:1403.1687, 2014.

34.

Chang

et al., End-to-End ASR with Adaptive Span Self-Attention, in: INTERSPEECH, 2020, pp. 3595–3599.

35.

Zhang

Loweimi

Bell

and Renals

, Stochastic attention head removal: A simple and effective method for improving transformer based asr models, arXiv preprint arXiv:2011.04004, 2020.

36.

Zhang

Loweimi

Bell

and Renals

, On the usefulness of self-attention for automatic speech recognition with transformers, in: 2021 IEEE Spoken Language Technology Workshop (SLT), IEEE, 2021, pp. 89–96.

37.

Perslev

et al., U-time: A fully convolutional network for time series segmentation applied to sleep staging, Advances in Neural Information Processing Systems 32 (2019).

38.

Ronneberger

Fischer

and Brox

, U-net: Convolutional networks for biomedical image segmentation, in: Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III 18, Springer, 2015, pp. 234–241.

39.

Jiang

et al., Nextformer: A ConvNeXt Augmented Conformer For End-To-End Speech Recognition, arXiv preprint arXiv:2206.14747, 2022.

40.

et al., Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline, in: 2017 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment (O-COCOSDA), IEEE, 2017, pp. 1–5.

41.

Fan

et al., Multiscale vision transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6824–6835.

42.

Zhang

Ren

and Sun

, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.

43.

et al., Improved multiscale vision transformers for classification and detection, arXiv preprint arXiv:2112.01526, 2021.

44.

Simonyan

and Zisserman

, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556, 2014.

45.

Roy

A.G.

Navab

and Wachinger

, Concurrent spatial and channel ‘squeeze & excitation’ in fully convolutional networks, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2018: 21st International Conference, Granada, Spain, September 16–20, 2018, Proceedings, Part I, Springer, 2018, pp. 421–429.

46.

Park

D.S.

et al., Specaugment on large scale datasets, in: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2020, pp. 6879–6883.

47.

Park

D.S.

et al., Specaugment: A simple data augmentation method for automatic speech recognition, arXiv preprint arXiv:1904.08779, 2019.

48.

Kudo

and Richardson

, Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, arXiv preprint arXiv:1808.06226, 2018.

49.

Kingma

D.P.

and Ba

, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.

50.

Hermans

J.R.

Spanakis

and Möckel

, Accumulated gradient normalization, in: Asian Conference on Machine Learning, PMLR, 2017, pp. 439–454.

51.

Pan

et al., ASAPP-ASR: Multistream CNN and self-attentive SRU for SOTA speech recognition, arXiv preprint arXiv:2005.10469, 2020.

52.

Han

K.J.

Prieto

and Ma

, State-of-the-art speech recognition using multi-stream self-attention with dilated 1d convolutions, in: 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), IEEE, 2019, pp. 54–61.

53.

Wang

et al., Transformer-based acoustic modeling for hybrid speech recognition, in: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2020, pp. 6874–6878.

Sampleformer: An efficient conformer-based Neural Network for Automatic Speech Recognition

Abstract

Keywords

1. Introduction

3. Methods

3.1 Macro-architecture design

3.2.1 Convolutional downsampling module

4.1 Data and training setup

Data

Training setup

Table 1 Detailed architecture configurations for Sampleformer and Conformer-CTC (baseline)

Table 4 Ablation study on group size parameters

Footnotes

Acknowledgments

References

Table 1
Detailed architecture configurations for Sampleformer and Conformer-CTC (baseline)

Table 4
Ablation study on group size parameters