Automatic Speech Recognition for Air Traffic Control Communications

Abstract

A significant fraction of communications between air traffic controllers and pilots is through speech, via radio channels. Automatic transcription of air traffic control (ATC) communications has the potential to improve system safety, operational performance, and conformance monitoring, and to enhance air traffic controller training. We present an automatic speech recognition model tailored to the ATC domain that can transcribe ATC voice to text. The transcribed text is used to extract operational information such as call-sign and runway number. The models are based on recent improvements in machine learning techniques for speech recognition and natural language processing. We evaluate the performance of the model on diverse datasets.

Air traffic controllers play a critical role in ensuring safe separation of aircraft in the airspace as well as on the airport surface. Despite the increasing deployment of datalink technologies, a significant fraction of communications between air traffic controllers and pilots is through speech, via radio channels. The automatic transcription of air traffic control (ATC) communications has many potential applications such as improving system safety, operational performance, and conformance monitoring, and enhancing controller training ( 1 ). However, automatic speech recognition (ASR) systems proposed to date for the ATC applications have not yet demonstrated the levels of accuracy needed for practical deployment ( 2 ).

Factors such as noisy radio channels, high speech rates, and diverse accents pose challenges to the development of ASR systems for ATC. On the other hand, ATC communications contain domain-specific vocabulary and standard phraseology that can be leveraged to tailor algorithms. Recent developments in machine learning such as deep neural networks have led to more accurate speech recognition algorithms ( 3 ), and suggest the possibility of better ASR algorithms for ATC communications.

Background

Related Work

ASR techniques have several potential applications in the ATC domain, including safety monitoring of live operations ( 4 – 6 ) and identification of anomalous aircraft trajectories ( 4 ). In another example of safety monitoring, Chen et al. ( 5 ) developed a framework to automatically flag pilot–controller miscommunications (read back error detection) to prevent untoward incidents. With the introduction of electronic flight strips in ATC towers, speech assistants have been shown to reduce air traffic controller workload in early demonstrations ( 7 ). ASR systems can also be used instead of pseudopilots for training air traffic controllers ( 8 , 9 ), and for human-in-the-loop simulations and workload measurements in air traffic management research ( 10 , 11 ). Additionally, the increasing demand for unmanned aerial vehicle operations has stimulated the need for ASR systems ( 12 ). However, the lack of sufficiently accurate ASR models for the ATC domain has remained a significant barrier to its deployment in these applications.

The earliest ASR models for large vocabularies used hidden Markov models (HMM) ( 13 ). Later approaches used hybrid models—a mixture of HMMs with Gaussian mixture models (GMM) or deep neural networks (DNN) ( 14 , 15 ). Recently, end-to-end speech recognition using deep neural networks has yielded significant improvements in accuracy for regular conversational speech ( 3 ). A key benefit of the end-to-end speech recognition model over classical approaches such as HMM-based models is the ease of training, since they do not require complicated pipelines with extensively engineered processing stages. There are several open-source ASR toolkits that allow researchers to adapt various models to a domain of interest: Bavieca ( 16 ), CMU Sphinx ( 17 ), HMM toolkit ( 18 ) and Kaldi ( 19 ) for HMM-based ASR models, and Deep Speech ( 20 ), wav2letter++ ( 21 ) and Jasper ( 22 ) for deep-learning-based end-to-end speech recognition models.

ATC applications of ASR have traditionally used HMM-based approaches, with limited consideration of end-to-end speech recognition approaches ( 6 ). In the recently concluded Airbus ATC speech recognition challenge, the top performing team used an HMM-based hybrid model that yielded a word error rate (WER) of 7.6% on the Airbus test data ( 23 ), with the second ranking team using an end-to-end speech recognition architecture that yielded a WER of 8.4% ( 24 ). In another recent study, a comparison of hybrid model (HMM/DNN in Kaldi) and end-to-end speech recognition for ATC voice in English and Chinese was evaluated by Lin et al. ( 6 ). For English ATC voice, they found that the end-to-end speech recognition model yielded a WER of 6%, performing better than the hybrid model that yielded a WER of 9%. One needs to note that the performance of the models, in WER, also depends on the quality of test data. If the test data is more representative of the training data and is less noisy, one expects better performance from the speech recognition model. Therefore, one cannot compare models based on WER if they are reported on different test sets.

The accuracy of a speech recognition model depends on the amount of labeled training data available. The amount of publicly available transcribed ATC voice data is relatively small compared with thousands of hours of transcriptions available for regular conversational speech (used for training commercial speech recognition models). To tackle the issue of limited data availability, Srinivasamurthy et al. ( 25 ) proposed using semi-supervised learning, in which new transcripts generated from a preliminary trained model were used to retrain the model, resulting in a 25% reduction in the WER. Researchers have also used flight trajectories to add context to a speech recognition model (e.g., flagging a flight as about to land) to improve its accuracy. Such a context-aware ASR model has been shown to reduce the command error rate (a measure of extracting commands from the transcripts) by 50%, although there were no significant improvements in the WER ( 26 ). Prior work has also leveraged the smaller vocabulary and standard phraseology of ATC communications to develop better language models, thereby achieving up to 20% improvement in the accuracy of the speech recognition model ( 27 ).

Contributions of This Paper

In this paper, we develop an ASR model that transcribes ATC voice communications to text. The proposed model is based on an end-to-end speech recognition architecture with a deep neural network, which offers several advantages over traditional HMM-based approaches that have been typically used for the ATC domain. HMM-based models typically contain multiple modules (acoustic model, pronunciation model, etc.), with each module being independently optimized with its own objective function which does not guarantee optimality across modules ( 28 ). By contrast, an end-to-end model replaces multiple modules with a single deep neural network that enables direct mapping of the acoustic signals to a sequence of characters, without hand-engineered intermediate states. As a result, one could attain optimality over the entire pipeline with end-to-end models by merging multiple modules into one optimized deep neural network and designing objective functions that truly reflect the final evaluation criterion. Moreover, an end-to-end modeling approach is easier to train, and has been shown to yield a better accuracy than traditional methods for conversational speech ( 3 ). In addition to training the model with ATC voice transcripts, we compare the model accuracy with transfer learning and parameter fine-tuning using a model that is pre-trained on regular conversational speech. Since the accuracy of the speech recognition model depends significantly on the amount of training data, we have compiled an extensive ATC speech corpus from various sources to understand the impact of using diverse ATC voice datasets for training the model.

Most applications require the extraction of information from the transcripts once the ATC voice data has been transcribed. For this purpose, we present a methodology to accurately extract operational information such as aircraft call-signs and runway assignments from transcripts, using state-of-the-art techniques from natural language processing. We also propose a performance metric called normalized uncertainty score to evaluate the accuracy of transcription in the absence of ground truth, a necessity for making decisions using transcribed ATC voice data.

Automatic Speech Recognition Model

The ASR model is based on Deep Speech ( 20 ), an end-to-end speech recognition model. We use Mozilla’s implementation of Deep Speech for our analysis ( 29 ). We briefly describe the model in this section; more detailed information can be found in the original papers ( 20 , 29 ).

Model Overview

Figure 1 illustrates the model architecture of the speech recognition system. The main components are the feature extraction module, acoustic model, language model, and decoding module. The feature extraction module takes the ATC audio signal as input, and outputs coefficients associated with its frequency spectrum as follows: the entire time series of the audio signal is divided into smaller (32 ms) time-windows with a 20 ms overlap; each time-window is associated with a feature vector that corresponds to its mel frequency cepstral coefficients (MFCCs) ( 29 ). The MFCCs of each time-window constitute the feature vector that serves as the input to the acoustic model. The acoustic model is a recurrent neural network that is trained to output a sequence of character probabilities based on the sequence of input feature vectors. The characters here correspond to letters of the English alphabet, apostrophe, space, blank, and additional identifiers corresponding to foreign words or unintelligible words. The language model outputs the probability of a sequence of characters based on the training text data, independent of the audio signal. In the decoding module, the output of the acoustic model is integrated with that of the language model to determine the transcribed text.

Figure 1.

Model architecture for the automatic speech recognition (ASR) system.

Acoustic Model

The acoustic model is a recurrent neural network (RNN) composed of five hidden layers, the first three of which are non-recurrent. The first layer takes the MFCC input for a particular time-window as well as a few (nine, in our case) context frames on either side of the window. A clipped rectified-linear unit (ReLU) activation function is used for all the layers, except the last one. The model architecture of Mozilla’s implementation differs slightly from the original Deep Speech paper ( 20 , 29 ). Here, the fourth layer is a feed forward recurrent layer instead of a bidirectional recurrent layer, to reduce computational time during inference. The last layer is a non-recurrent output layer that yields the character probabilities based on a Softmax function. A connectionist temporal classification (CTC) loss function is used to compute the prediction error during training ( 30 ). We use an adaptive learning rate (Adam method) for updating the model parameters through stochastic gradient descent during training, and a dropout rate of 5% for regularization.

Language Model

Training the acoustic model for the ATC domain to produce accurate character level transcription is challenging because of the limited availability of transcribed ATC audio. Furthermore, the neural network can output phonetically similar word renderings which can be incorrect (e.g., “bostin” instead of “boston”). These issues can be addressed using a language model that is tailored to the ATC domain. The language model is a probability distribution over a sequence of words, which is used in the decoding stage. We choose an N-gram language model because it can be trained using existing libraries (KenLM [31]) and yields good performance. In this language model, the probability of the $k t h$ word is assumed to depend only on the $N - 1$ preceding words, that is, $P (w_{k} | w_{k - 1}, w_{k - 2}, . ., w_{1}) = P (w_{k} | w_{k - 1}, w_{k - 2}, . ., w_{k - (N - 1)})$ . Consequently, the probability of a sequence of words, $P (w_{1}, w_{2}, . . . w_{m})$ , can be expressed as a product of conditional probabilities.

The conditional probabilities required for the model are determined from the ATC audio transcripts in the training data. Although a higher value of $N$ will lead to better predictions, there may not be sufficient data to obtain consistent statistics for a larger $N$ . An optimal value of $N$ is determined using a parametric analysis to yield a lower WER on the validation set.

Decoding Step

The decoding step determines the most probable sequence of characters given the output probabilities from the acoustic model and language model. Let $P_{A} (c | x)$ represent the probability of a sequence of characters, $c = {c_{1}, c_{2}, c_{3} . .}$ , obtained from the acoustic model for a given input audio ( $x$ ). Similarly, let $P_{LM} (c)$ represent the probability of a sequence of characters obtained from the language model. The objective of the decoding step is to obtain a sequence of characters that maximizes the confidence score, $Q (c)$ :

Q (c) = P_{A} (c | x) + α P_{LM} (c) + β word count (c).

(1)

Here, $α$ and $β$ are weights to balance the influence of the acoustic model, the language model, and the word count of the utterance. The last term is used because shorter sentences inherently have a higher probability in the N-gram language model. The output sequence of characters that maximizes the confidence score, $Q ()$ , is determined using a beam search algorithm ( 20 ). Optimal values of $α$ , $β$ , and beam width are determined using a parametric analysis such that the average WER over a validation set is minimized.

Model for ATC Communications

Data Sources

Training an ASR model requires large amounts of transcribed audio data. The datasets used in our study are shown in Table 1, and include transcripts of ATC communications from the U.S. and Europe, with varied accents and audio quality. Most of the data corresponds to approach and tower control segments. The European datasets also contain utterances that are partially non-English. Those parts, as well as non-intelligible parts of the audio, are encoded with a unique identifier in the transcripts. The total number of hours of audio transcription is approximately 140 h, yielding 84 h after removing outliers and silent portions of audio. To the best of our knowledge, this is the first study that combines these four diverse datasets.

Table 1.

Available Transcribed ATC Voice Datasets

Corpus name	Duration	Comments
AIRBUS-ATC ( 23 )	40 h	French-accent speech; approach and tower control, ATIS.
ATCOSIM Corpus ( 32 )	10 h	German-Swiss-French accent; en-route controllers’ speech recorded in a studio environment.
ATC Communication ( 33 )	20 h	Ground control, tower control, approach control, and area control. Primary data source from Czech Republic’s ANSP, with small amounts from Lithuania and Philippines.
NIST ATC Complete (LDC94S14A) ( 34 )	70 h	ATC data for approach control from three U.S. airports (BOS, DCA, DFW).

Note: ATC = air traffic control; ATIS = Automatic Terminal Information Service; ATCOSIM = Air Traffic Control Simulation Speech; ANSP = Air Navigation Service Provider; NIST = National Institutes of Standards and Technology; BOS = Boston Logan International Airport; DCA = Ronald Reagan Washington International Airport; DFW = Dallas/Fort Worth International Airport.

Performance Metrics

The WER is a widely used measure of ASR model accuracy. It is defined as the sum of the number of words in the transcribed text that are either substituted ( $S$ ), deleted ( $D$ ), or inserted ( $I$ ) relative to the reference text, divided by the number of words in the reference ( $N_{r}$ ):

WER = \frac{S + D + I}{N_{r}} .

(2)

The WER includes errors arising from filler words or other words that are not relevant in a particular context. The concept error rate (CER) is an alternative metric that reflects the accuracy of domain-specific ASR systems. The CER is given by number of misrecognized concepts divided by the total number of concepts. This study focuses on the WER primarily because the number of concepts in the speech that one would be interested in (for evaluating the CER) depends on the application area. Additionally, one needs the labels of the concepts in the training data to evaluate the CER, which is not available in our dataset.

The speed of transcription is also an important quantity. It is measured as the real-time factor (RTF), namely, the duration of input audio divided by the required time to process the input. Lower RTF, which implies faster transcription speed, is preferred. However, RTF might not be significantly important for off-line applications that involve accurately transcribing recorded voice datasets. On the other hand, if one is interested in transcribing the speech in real time (in applications such as safety monitoring), then one needs to have RTF less than one. The RTF for our models is around 0.3, implying that they can be deployed in live operations.

Model Variants

We consider the following four model variants in our study:

Model-1: Only an Acoustic Model and No Language Model

Here, the transcription is based only on the acoustic model, and the language model weights $(α, β)$ are set to zero. This variant serves as a baseline to assess the benefit of including a language model. The number of neurons in each hidden layer of the neural network was fine-tuned based on a parametric analysis that involved varying its value in the range (500,1024) and picking the optimal value that yielded the lowest average WER on the validation set. The optimal number was found to be 650 neurons in each hidden layer. We note that the original Deep Speech model ( 20 ) had a significantly higher number of neurons in each hidden layer (2,048 neurons). However, we found that a smaller number of neurons was optimal because of the smaller amount of training data available relative to conversational speech. In addition to English characters, some of the datasets use special characters to represent foreign words (@), non-intelligible words (_), breaks (/) and noise (#). We include these special characters in our training data.

Model-2: Acoustic Model and Language Model

The language model weights $(α, β)$ are chosen by sweeping through different values of $(α, β)$ , and choosing the set of parameters that yields the lowest WER over a validation set. Figure 2 shows the computed WER on the validation set for different values of $(α, β)$ . Using this, we determined the parameters that yield the lowest WER. A similar parametric analysis resulted in a choice of $N = 4$ for the N-gram language model. Ideally, one should search for the optimal set of parameters over the joint-space of the hyperparameters. However, to limit the number of computations, we individually optimized the beam width, the N-value of the language model, and the number of neurons, while keeping the other parameters fixed at their nominal values.

Figure 2.

Word error rate of validation set for various language model weights $(α, β)$ .

Model-3: Refining Parameters of a Pre-Trained Model

We used an acoustic model that was trained for regular conversational speech, and refined the neural network parameters using the ATC speech corpus. The pre-trained model is a much larger neural network with 2,048 neurons within each layer. The pre-trained acoustic model was developed using multiple speech corpora (3,250 h with Fisher, LibriSpeech, and Switchboard datasets) and achieves a WER of 0.08 on conversational speech (LibriSpeech clean test corpus) ( 29 ). The neural network parameters of the pre-trained model were optimized by retraining the model using the ATC speech corpus. The optimization was conducted over four epochs using a small learning rate ( $10^{- 5}$ ) for the gradient descent. The choice of four epochs was based on common practice, and can be further revised considering the performance over the validation set. As before, we used a language model that was trained using the ATC corpus.

Model-4: Transfer Learning with a Pre-Trained Model

Transfer learning has shown promise in machine learning applications where there is limited domain-specific labeled training data available ( 35 ). The idea is to adapt a pre-existing trained model to the domain of interest. In our case, we adapted a pre-existing acoustic model that is trained for regular conversational speech (the same baseline model used for Model-3) to the case of ATC communications. The most common approach to transfer learning with neural networks is to “freeze” the parameter values of a few layers of a trained neural network, and to retrain the parameters of the other layers using domain-specific data. For ASR, the parameters of the first few layers are frozen from the pre-trained model, and parameters of the remaining layers are trained using the ATC speech corpus. The rationale for such an approach is that the first few layers represent some form of filtering or feature extraction that might be common across different application domains. This methodology has been used to train ASR models for different languages using a pre-trained model for the English language ( 36 ). In our study, we froze the parameters of the first three layers of a pre-trained ASR model for conversational speech (from the previous variant), and retrained the parameters of the last two layers using the ATC speech corpus.

Model Performance

We first trained and tested the different model variants with just one of the datasets (AIRBUS-ATC), and then included the other datasets. The initial analysis with a single dataset allowed us to test different model variants without significant computational effort. We used approximately 36 h of the 40-h AIRBUS-ATC dataset for training the model, 2 h for validation, and 2 h for testing.

A summary of the WERs of the four models (on a 2-h test dataset with 1,500 utterances) is shown in Table 2. The acoustic model integrated with the language model (Model-2) performs the best among the four approaches with an average WER of 0.22. This is followed by the model obtained by fine-tuning a pre-trained ASR model (Model-3). By comparing results from Model-2 and Model-1, we notice that integrating the language model with the acoustic model improves the speech recognition accuracy by about 18%. Transfer learning yields the lowest accuracy among the different approaches, possibly because the higher speech rates and noise of ATC audio require significantly different parameter values in the first few layers of the neural network compared with a model trained on conversational speech.

Table 2.

WERs and $\bar{WER} s$ on the AIRBUS-ATC Test Set for Different Model Variants

Variant	Model type	WER	$\bar{WER}$
Model-1	Acoustic model without language model	0.27	0.23
Model-2	Acoustic model integrated with language model	0.22	0.17
Model-3	Parameter fine-tuning with pre-trained model	0.25	0.23
Model-4	Transfer learning with pre-trained model	0.54	0.52

Note: WER = word error rate; $\bar{WER}$ = adjusted word error rate.

Figure 3a shows a box-plot of the average WER as a function of the word count in the utterance. The WER is seen to be higher for utterances with a lower word count, and is nearly constant for word counts greater than six. The reason for this behavior is that missing one or two words (even filler words such as “um” or “ah”) would lead to a higher WER for smaller sentences. Figure 3b shows the histogram of the word count of utterances in the test set. The average word count is 12 words, and utterances with fewer than six words represent 7% of the test set. Additionally, 10.3% of the utterances in the test set have a foreign word or non-intelligible word in the transcription. We therefore compute the WER after excluding utterances with fewer than six words and the ones having foreign words or non-intelligible words, and denote it by $\bar{WER}$ . One needs to note that shorter sentences might be important depending on the application (such as detecting acknowledgments or readback errors). For applications such as extracting operational information, the newly defined performance metric, $\bar{WER}$ , might be of interest because such information is primarily contained in longer sentences. The $\bar{WER}$ of the four model variants is shown in Table 2. We note that $\bar{WER}$ is 8% to 25% lower than the WER. Overall, Model-2, wherein the acoustic and language models are trained using only ATC domain-specific speech data, performs the best among these approaches, with $\bar{WER}$ equal to 0.17.

Figure 3.

Statistics of word counts and word error rates (WERs) computed for the test set using Model-2: (a) WER for different word counts and (b) histogram of word count.

As one might expect, it takes more time to train a neural network model from scratch (Models-1,2) than it takes for parameter fine-tuning (Model-3), in which the optimization is conducted over a smaller number of epochs (four in our case), or even transfer learning (Model-4) in which only the last two layers of the neutral network are retrained. However, we use fewer neurons for training the model from scratch compared with the pre-trained model, which slightly reduces the computational time (however, it is still higher than the other two methods).

Training with Additional Datasets

To understand the benefit of additional training data, we added utterances from the three other corpora—ATCOSIM (Air Traffic Control and Simulation Speech) Corpus, ATC Communication, and NIST (National Institute of Standards and Technology) ATC Complete—to the AIRBUS-ATC data, and retrained the acoustic model. The training set consisted of about 65,500 utterances (80 h of non-silenced audio) of ATC conversations. A comparison between the WER computed using the newly trained model with additional data and the earlier model, evaluated on the AIRBUS-ATC and NIST ATC Complete (100 utterances) testsets, is shown in Table 3. The $\bar{WER}$ on the AIRBUS-ATC test set increases from 0.17 to 0.29 with additional data. A possible reason for this increase is that the datasets are diverse with regards to the accents, audio quality, and navigational information. However, $\bar{WER}$ evaluated on the NIST ATC Complete dataset reduces from 0.75 to 0.32, illustrating that training with additional data increases the generalizability of the model. These comparisons use the same language model (trained using AIRBUS-ATC data) even for the evaluation with the NIST ATC data, which contributes to the lower accuracy on that test set. We also retrained the model that was initially trained with all the four datasets by parameter fine-tuning with the AIRBUS-ATC training set (similar to the approach explained in the previous section). The results indicate that parameter fine-tuning reduces the WER on the AIRBUS-ATC test set (Table 3) compared with the case without fine-tuning, but still yields lower accuracy than when the sources of the training and test sets are the same (i.e., AIRBUS-ATC).

Table 3.

Effects of Using Additional Training Data Sources

Training data source	$\bar{WER}$ on test set
	AIRBUS-ATC	NIST ATC Complete
AIRBUS-ATC training set	0.17	0.75
All data sources	0.29	0.32
All data sources + fine-tuning with AIRBUS-ATC training set	0.19	0.45

Note: $\bar{WER}$ = adjusted word error rate; ATC = air traffic control; NIST = National Institute of Standards and Technology.

Extracting Operational Information

ATC communications contain extensive operational information (e.g., runway assignment, heading, flight level) that are of value in decision-making. In this section, we present a methodology to automatically extract operational information from ATC communication transcripts. Natural language processing (NLP) techniques for extracting information from unstructured text can be broadly classified into statistical models and rule-based grammar. Statistical models are obtained using machine learning algorithms on the text data. On the other hand, rule-based grammar techniques are based on a collection of hand-engineered rules to perform the NLP task. We employ both the methodologies in this paper: a statistical model is used for call-sign extraction, and a rule-based grammar approach is used to extract runway information.

Call-Sign Extraction

The utterances of the air traffic controllers and the pilots often contain the aircraft call-sign, which serves as a unique identifier for a flight receiving or sending the information. The call-sign typically comprises the airline telephony designation (such as “American” for “American Airlines”; “Speedbird” for “British Airways”), followed by the flight ID, which consists of two to four digits, with an optional suffix of two to three letters. A few examples of ATC utterances are shown below, with the aircraft call-sign highlighted:

air nostrum eight seven six one contact bordeaux one three three decimal seven seven five goodbye

five thousand feet one zero one zero easy six eight four romeo

lufthansa zero one charlie reduce speed two two zero knots

To associate a flight with the instruction or command given by the air traffic controller or pilot, one must be able to accurately extract call-signs from the transcript. There are multiple challenges to accurate call-sign extraction: (a) controllers could use multiple airline identifiers for the same airline; (b) the number of words following the airline identifier (corresponding to the numerals/NATO phonetics) can vary; and (c) there could be call-signs with just the flight number without the airline identifier. Although the digits are expected to be pronounced as separate numbers as per ICAO (International Civil Aviation Organization) nomenclature, the convention can vary in different regions (for example, in the U.S., “2020” is often spoken as “twenty twenty” instead of “two zero two zero”). One could also have non-standard pronunciation: for example, “0” could be pronounced as “oh,”“9” could be pronounced as “nine” instead of “niner,” and “F” could be pronounced as “fox” instead of “foxtrot.”

Given these variations, a rule-based grammar approach for call-sign extraction might not be feasible. We instead use a statistical approach to train a model to identify call-signs using transcripts of ATC communications. To extract call-signs, we use named entity recognition (NER), a standard NLP technique, to classify information within unstructured text into predefined categories. For the call-sign extraction problem, the objective is to identify a sequence of words in each utterance, and to categorize it as the call-sign, if it exists. We use a Python library called Spacy, which is one of many standard libraries for NER ( 37 ). The NER model in Spacy is based on a deep convolution neural network and uses sub-word features. To categorize a particular word, the model accounts for the neighboring three words on either side. A key advantage of the sub-word feature is that it can identify call-signs even if there are spelling errors in the transcripts, a particularly useful property for ASR-generated transcripts.

Performance Metrics

The accuracy of the model is evaluated using three performance measures: precision, recall, and F-1 score. We introduce some notation to define these performance measures: Let $A$ be the number of instances in the test set when there is no call-sign in the reference text, and the model also detects no call-sign. Similarly, let $B$ be the number of instances when there is a call-sign in the reference text, but the model does not detect a call-sign. Let $C$ be the number of instances when there is no call-sign in the reference text, but the model erroneously detects a call-sign. Let $D$ and $E$ be the number of instances when the detected call-sign is correct and incorrect, respectively, among the instances when the reference text has a call-sign. The mathematical expressions for precision ( $P$ ), recall ( $R$ ), and F-1 score are as follows:

P = \frac{D}{C + D + E}; R = \frac{D}{B + D + E}; F - 1 = \frac{2 P R}{P + R} .

(3)

Precision is the fraction of instances for which the call-sign is correctly extracted from among the instances when a call-sign is extracted. Recall is the fraction of instances for which the correct call-sign is extracted from among the instances when the call-sign is present in the reference text. The F-1 score is the harmonic mean of precision and recall.

Results

The model was trained using 25,000 samples of labeled data, which contained the transcript of the utterance, and the corresponding call-sign in the utterance (if present). The independent test data set contained approximately 3,000 utterances. For the call-sign extraction task from the reference transcript, the precision was 0.97, the recall was 0.93, and the F-1 score was 0.95. In other words, 97% of the call-signs that were extracted matched the reference call-sign, and 93% of the call-signs in the reference text were extracted correctly. Table 4 shows a few example transcripts of the utterances in the test set, along with the reference call-sign contained within the transcript, and the extracted call-sign from the model. The first three rows in the table are instances in which the call-sign was correctly extracted. The next three rows illustrate cases wherein the call-sign present in the text was not extracted, which would affect the recall score. In these cases, the primary reason for the error was that either the airline identifier in the test set was very different from those seen in the training set, or that the call-sign did not follow the usual convention. The last row in the table shows a case in which the model incorrectly extracted a call-sign when there was none in the reference transcript, which would affect the precision score. Overall, a F-1 score of 0.95 is a good initial step, and can be improved by increasing the amount of training data. One needs to note that even if the call-sign was not seen in the training data, the model is capable of correctly identifying the call-sign based on neighboring words that gives it context. Table 5 shows the performance measures for the call-sign extraction task on the test set for different amounts of data in the training set. The performance of the model improves as the amount of training data increases, but with diminishing marginal returns, as one would expect.

Table 4.

Extracting Call-Signs from Reference Transcripts

Reference call-sign	Reference transcript	Extracted call-sign
Easy four seven tango lima	Roger we call you huh short final easy four seven tango lima	Easy four seven tango lima
Beeline four papa golf	Report huh short final maximum one sixty beeline four papa golf	Beeline four papa golf
Swiss one juliet bravo	Keep rolling and vacating via mike eight swiss one juliet bravo	Swiss one juliet bravo
Easy _ whiskey	Toulouse tower @ easy _ whiskey established on the ils three two right	NA
Airbus delta sierra	Airbus delta sierra after takeoff you will maintain runway axis	NA
Binair seven alpha	# Huh good huh @ huh binair seven alpha approaching november eight	NA
NA	We leave via sierra three huh india lima	Sierra three huh india lima

Note: NA= not available.

Table 5.

Call-Sign Extraction Performance for Different Amounts of Training Data

Training dataset size (# of utterances)	Precision	Recall	F-1 score
100	0.91	0.34	0.50
500	0.95	0.49	0.65
5,000	0.99	0.90	0.94
25,000	0.97	0.93	0.95

Next, we look at the performance of the call-sign extraction task on the transcribed voice data that is output from the speech recognition model. These results are based on the transcripts generated from an ASR model with a $\bar{WER}$ of 0.17. For the call-sign extraction task from the transcribed voice data, the precision is 0.81, recall is 0.57, and the F-1 score is 0.67, on an independent test set. The performance here is worse than in the case where we had the actual transcripts, because of inaccuracies in the voice transcription. It is worth noting that even when the transcription of voice is not completely accurate (as reflected by the WER), it may be possible to accurately extract the call-sign from the transcript if the call-sign part of the utterance is transcribed correctly.

Some limitations of this statistical approach to call-sign extraction must be noted: (a) detecting airlines unseen in the training set can be challenging, even though the approach considers neighboring words for context; (b) call-sign digits are pronounced differently in the U.S. and in Europe. These limitations can be overcome by including more diverse data sources for training.

Extracting Runway Information

A rule-based grammar was used for extracting runway information from the transcript. Reasons for employing a rule-based grammar approach rather than a statistical approach for runway extraction include: (a) the datasets do not contain the labeled runway information needed to build a statistical model; and (b) relatively few runway numbers are uttered in the datasets (because the voice recordings are from a small number of airports), which is not sufficient to obtain a statistical model that generalizes well. Furthermore, the utterances of runway numbers are highly structured, motivating the use of rule-based grammar. Runway numbers are uttered in the following manner: one or two digits followed by “left,”“right,” or “center.” The transcript is searched for such patterns to obtain the runway information for extraction. We hand-labeled about 200 utterances to test the performance of a rule-based grammar approach for extracting runway information. The precision, recall and F-1 score is 1 on the reference transcript, indicating that the rule-based approach is perfectly accurate in extracting runway information. Runway information extraction using the transcribed text (from the ASR model with $\bar{WER}$ of 0.17) shows very good performance, achieving a precision of 0.97, recall of 0.93, and F-1 score of 0.95. Note that for airports with a single runway (which were not present in the data used for this analysis), runway numbers might not include “left,”“right,” or “center” modifiers. In such cases, one needs to appropriately modify the rule-based grammar by including “runway” followed by a digit as a keyword identifier.

Extensions and Opportunities for Further Research

In this section, we briefly discuss some promising extensions and directions for further investigation.

Evaluating Transcription Accuracy in the Absence of Ground Truth Data

Calculation of the WER requires the availability of ground truth data (i.e., the actual transcript). However, the practical deployment of decision-making using ASR requires an estimate of the likely accuracy of the transcription, even in the absence of ground truth data. In other words, it is valuable to know when the ASR models are expected to be accurate, and when they are not. To this end, we propose to use the uncertainty score, defined as the negative logarithm of the confidence score, $Q (c)$ , as a surrogate for the WER. The confidence score of an utterance, and therefore its uncertainty score, depends on the character probabilities of the transcribed text, as determined by the acoustic model and language model of the ASR system. Figure 4a shows a scatter plot of the uncertainty score and WER, for each utterance in the test set. For the purpose of illustration, the uncertainty score was computed using only the acoustic model. The figure shows a clear correlation between the uncertainty score and the WER. This correlation increases further when one considers the normalized uncertainty score, defined as the uncertainty score divided by the number of characters in the transcription of the utterance. Figure 4b shows a scatter plot of the normalized uncertainty score and the WER, for the utterances in the test set. The normalized uncertainty score increases with the word error rate, and can be used to identify instances in which one would expect a high WER (i.e., low accuracy of the ASR model).

Figure 4.

Uncertainty score of the transcription: (a) uncertainty score, (b) normalized uncertainty score, and (c) distribution of normalized uncertainty score.

For applications in which the ASR model is used as part of a decision-support tool, one could flag, or even exclude, transcriptions that have a higher-than-average normalized uncertainty score. The uncertainty score can also be evaluated at the word level, when the accuracy of the extracted operational information is a quantity of interest. In addition, the uncertainty score can be used to identify off-nominal events during which we expect the performance of the speech recognition model to degrade (for example, because the speech rate or the phraseology are significantly different from the nominal periods over which the model was trained).

Figure 4c shows the distribution of the normalized uncertainty score computed over two test datasets: (a) one that is similar to the dataset used for training the ASR model (shown in blue, and representative of conversations during nominal events); and (b) a test dataset that is significantly different (in accent and speech rate) from the training data set (shown in red, and representative of conversations during off-nominal events). We observe that the distribution of the normalized uncertainty score for the off-nominal data is significantly different (with a higher mean value, indicative of higher WERs) from the distribution obtained for the test dataset that is similar to the training data. Hypothesis testing techniques can be used to determine if an utterance (or a series of utterances) lies outside the nominal distribution in a statistical sense, and to flag those periods as off-nominal events.

Alternative Language Models

In this paper, we used a standard word N-gram language model for simplicity, and focused primarily on the acoustic model and the extraction of operational information. However, prior studies have shown that more sophisticated language models such as the class N-gram or RNN-based language models can improve the accuracy of ASR systems ( 6 , 27 ). Another potential benefit of the class N-gram model is that one can efficiently incorporate out-of-vocabulary data, such as new airlines or navigational aids that are specific to a particular region, by updating the class definition file.

Semi-Supervised Learning

The availability of sufficient amounts of high-quality data is a key factor in improving the performance of ASR models. Our experiments have shown that the use of diverse datasets for model training results in more general models. However, there are not many openly available corpora for ATC communications, and the manual transcription of ATC conversations requires significant effort. A different direction is to use semi-supervised learning, namely, to utilize widely available but untranscribed ATC audio data ( 38 ) to improve the accuracy of the ASR model. The proposed approach would be as follows: (a) train a preliminary model with the existing ATC speech corpora; (b) use this preliminary model to transcribe utterances of ATC voice communications that do not have transcripts; (c) evaluate the accuracy of each of the utterances using the uncertainty score as a metric; and (d) retrain the ASR model by including the utterances that yielded a low uncertainty score in the training data. Although semi-supervised learning for ATC applications has been previously attempted ( 25 ), we believe that the use of the normalized uncertainty score to select data for retraining models is a promising direction for further investigation. Semi-supervised learning techniques ( 39 ) have been recently shown to perform well for regular conversational speech even with small amounts of labeled training data; these techniques could be extended to the ATC domain.

Acoustic Model

In this paper, we have used Mozilla’s implementation of the Deep Speech model that considered MFCC as the input feature. One could explore other alternative input features. For example, filter bank energies are an attractive alternate option given that there have been studies that have shown that they perform better than MFCCs for robust speech recognition ( 40 ). Further, recent studies have shown that transformer-based architecture for the acoustic model yields better performance than conventional RNN, even for low resource languages ( 41 ). This makes the transformer-based model an attractive alternative for the ATC domain.

Another important aspect that one might have to consider is constraints on the model inference time depending on the application of interest. A larger neural network model (number of hidden layers and neurons in each layer) for the speech recognition system typically requires higher computational time for inference, resulting in higher RTF. On the other hand, a larger model might yield better accuracy. For a practitioner, it might be of interest to quantify this trade-off between RTF and accuracy, which is a potential direction for future research.

Conclusions

This paper investigated the use of an ASR model for ATC voice communications data, and for the extraction of operational information from the same. The model followed an end-to-end speech recognition architecture, and was based on a recent machine learning model that involves RNN (Deep Speech). We were able to obtain a WER of 0.17 with our speech recognition model, on an independent test set of ATC communications data. The analysis revealed that including an N-gram language model in addition to the acoustic model improved the accuracy by 26%, and that transfer learning and parameter fine-tuning with a model pre-trained on conversational speech did not improve the accuracy. The results illustrated that, as expected, the use of diverse data sources during training resulted in more generalizable models, that is, models with better accuracy on test datasets from different sources. Recent ASR models ( 6 , 24 ) have reported better performance in WER compared with our proposed model. However, the intent of our research was to understand the impacts of the language model, parameter fine-tuning, transfer learning, and training with additional data, and to consider the potential to extract operational information using NLP techniques.

To this end, we illustrated a call-sign extraction method using NER, which yielded an F-1 score of 0.95 on the actual transcript, and an F-1 score of 0.69 on the transcript generated by the ASR model. The runway information was extracted using a rule-based grammar, resulting in an F-1 score of 1 (perfect transcription) on the actual transcript, and 0.95 on the transcript generated by the ASR model. We believe that the accuracy of call-sign extraction would further improve with more training data and better ASR accuracy. Further, we identified opportunities to improve the accuracy of ASR models for the ATC domain, including employing better language models, semi-supervised learning, incorporating better priors or context, and training the model with larger quantities of transcribed ATC speech data. These enhancements could potentially improve transcription accuracy, enabling practical applications ranging from real-time safety monitoring to speech assistants for air traffic controllers.

Footnotes

Acknowledgements

The authors would like to thank Azreen Zaman for processing some of the datasets ( 32 – 34 ) used to train the models, and Nadia Dimitrova for helpful discussions on potential language models for ATC communications. We also would like to thank Airbus for providing their ATC speech corpus ( 23 ), and Dr Jim Glass for providing access to the NIST ATC Complete corpus ( 34 ). This work has greatly benefited from discussions with Monica Alcabin, Robert Samis, Jonathan Rein, and Larry Goldstein.

Author Contributions

The authors confirm contribution to the paper as follows: study conception and design: S. Badrinath, H. Balakrishnan; data collection: S. Badrinath; analysis and interpretation of results: S. Badrinath, H. Balakrishnan; draft manuscript preparation: S. Badrinath, H. Balakrishnan. All authors reviewed the results and approved the final version of the manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the ACRP Graduate Research Award Program.

ORCID iDs

Sandeep Badrinath

Hamsa Balakrishnan

References

Kopald

H. D.

Chanen

Chen

Smith

E. C.

Tarakan

R. M.

Applying Automatic Speech Recognition Technology to Air Traffic Management. In 2013 IEEE/AIAA 32nd Digital Avionics Systems Conference (DASC). IEEE, 2013, pp. 6C3–1.

Nguyen

V. N.

Holone

Possibilities, Challenges and the State of the Art of Automatic Speech Recognition in Air Traffic Control. World Academy of Science, Engineering and Technology, International Journal of Computer, Electrical, Automation, Control and Information Engineering, Vol. 9, No. 8, 2015, pp. 1940–1949.

Amodei

Ananthanarayanan

Anubhai

Bai

Battenberg

Case

Casper

Catanzaro

Cheng

Chen

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin. In International Conference on Machine Learning. 2016, pp. 173–182.

Subramanian

S. V.

Kostiuk

P. F.

Katz

Custom IBM Watson Speech-to-Text Model for Anomaly Detection Using ATC-Pilot Voice Communication. In 2018 Aviation Technology, Integration, and Operations Conference. 2018, p. 3979.

Chen

Kopald

Chong

Wei

Levonian

Read Back Error Detection Using Automatic Speech Recognition. In 12th USA/Europe Air Traffic Management Research and Development Seminar (ATM2017), Seattle, WA, 2017.

Lin

Deng

Chen

Zhang

Yang

A Real-Time ATC Safety Monitoring Framework Using a Deep Learning Approach. IEEE Transactions on Intelligent Transportation Systems, Vol. 21, No. 11, 2020, pp. 4572-4581, https://doi.org/10.1109/TITS.2019.2940992.

Helmke

Rataj

Mühlhausen

Ohneiser

Ehr

Kleinert

Oualil

Schulder

Klakow

Assistant-Based Speech Recognition for ATM Applications. In 11th USA/Europe Air Traffic Management Research and Development Seminar (ATM2015), Lisbon, Portugal, 2015.

Šmídl

Švec

Pražák

Trmal

Semi-Supervised Training of DNN-Based Acoustic Model for ATC Speech Recognition. In International Conference on Speech and Computer. Springer, 2018, pp. 646–655.

Tarakan

Baldwin

Rozen

An Automated Simulation Pilot Capability to Support Advanced Air Traffic Controller Training. In The 26th Congress of ICAS and 8th AIAA ATIO. 2008, p. 8897.

10.

H.-L.

Cheng

V. H.

Ballinger

Fong

Nguyen

Jones

Cowart

S. E.

A Speech-Enabled Simulation Interface Agent for Airspace System Assessments. In AIAA Modeling and Simulation Technologies Conference, 2015, p. 0148.

11.

Cordero

J. M.

Rodríguez

de Pablo

J. M.

Dorado

Automated Speech Recognition in Controller Communications Applied to Workload Measurement. In 3rd SESAR Innovation Days, Stockholm, Sweden, 2013.

12.

Lowry

Pressburger

Dahl

D. A.

Dalal

Towards Autonomous Piloting: Communicating with Air Traffic Control. In AIAA Scitech 2019 Forum. 2019, p. 2207.

13.

Rabiner

L. R.

A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE, Vol. 77, No. 2, 1989, pp. 257–286.

14.

Gales

Young

The Application of Hidden Markov Models in Speech Recognition. Now Publishers Inc, 2008.

15.

Morgan

Bourlard

Continuous Speech Recognition. IEEE Signal Processing Magazine, Vol. 12, No. 3, 1995, pp. 24–42.

16.

Anon. The Bavieca ASR Toolkit. http://www.bavieca.org/index.html. Accessed June 11, 2020.

17.

Lamere

Kwok

Gouvea

Raj

Singh

Walker

Warmuth

Wolf

The CMU SPHINX-4 Speech Recognition System. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2003), Hong Kong, Vol. 1. 2003, pp. 2–5.

18.

Anon. Hidden Markov Model Toolkit. http://htk.eng.cam.ac.uk. Accessed June 11, 2020.

19.

Povey

Ghoshal

Boulianne

Burget

Glembek

Goel

Hannemann

Motlicek

Qian

Schwarz

Silovsky

The Kaldi Speech Recognition Toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Understanding, Hilton Waikoloa Village, Big Island, Hawaii, IEEE Signal Processing Society, 2011.

20.

Hannun

Case

Casper

Catanzaro

Diamos

Elsen

Prenger

Satheesh

Sengupta

Coates

A. Y.

Deep Speech: Scaling Up End-to-End Speech Recognition. arXiv preprint arXiv:1412.5567.

21.

Pratap

Hannun

Cai

Kahn

Synnaeve

Liptchinsky

Collobert

Wav2letter++: The Fastest Open-source Speech Recognition System. CoRR, Vol. abs/1812.07625. https://arxiv.org/abs/1812.07625.

22.

Lavrukhin

Ginsburg

Leary

Kuchaiev

Cohen

J. M.

Nguyen

Gadde

R. T.

Jasper: An End-to-End Convolutional Neural Acoustic Model. arXiv preprint arXiv:1904.03288.

23.

Delpech

Laignelet

Pimm

Raynal

Trzos

Arnold

Pronto

A Real-Life, French-Accented Corpus of Air Traffic Control Communications. In Language Resources and Evaluation Conference (LREC), Miyazaki, Japan, 2018.

24.

Pellegrini

Farinas

Delpech

Lancelot

The Airbus Air Traffic Control Speech Recognition 2018 Challenge: Towards ATC Automatic Transcription and Call Sign Detection. arXiv preprint arXiv:1810.12614.

25.

Srinivasamurthy

Motlicek

Himawan

Szaszák

Oualil

Helmke

Semi-Supervised Learning with Semantic Knowledge Extraction for Improved Speech Recognition in Air Traffic Control. Proceedings of Interspeech 2017, Stockholm, Sweden 2017.

26.

Oualil

Klakow

Szaszák

Srinivasamurthy

Helmke

Motlicek

A Context-Aware Speech Recognition and Understanding System for Air Traffic Control Domain. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017, pp. 404–408.

27.

Nguyen

V. N.

Using Linguistic Knowledge for Improving Automatic Speech Recognition Accuracy in Air Traffic Control. Master’s thesis. Østfold University College, Halden, 2016.

28.

Wang

An Overview of End-to-End Automatic Speech Recognition. Symmetry, Vol. 11, No. 8, 2019, p. 1018.

29.

Mozilla Corporation. Deep Speech Model. https://deepspeech.readthedocs.io/en/r0.9/. Accessed May 7, 2021.

30.

Kawakami

Supervised Sequence Labelling with Recurrent Neural Networks. PhD dissertation. Technical University of Munich, 2008.

31.

Heafield

KenLM: Faster and Smaller Language Model Queries. Proc., 6th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 2011, pp. 187–197.

32.

Eurocontrol. https://www.eurocontrol.int/eec/public/standard_page/EEC_News_2008_1_ATCOSIM.html. Accessed September 9, 2019.

33.

Anon, University of West Bohemia, Department of Cybernetics. https://lindat.mff.cuni.cz/repository/xmlui/handle/11858/00-097C-0000-0001-CCA1-0?show=full. Accessed September 9, 2019.

34.

Godfrey

Air Traffic Control Complete LDC94S14A. Web Download. Philadelphia: Linguistic Data Consortium, 1994, 2020.

35.

Pan

S. J.

Yang

A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, Vol. 22, No. 10, 2009, pp. 1345–1359.

36.

Kunze

Kirsch

Kurenkov

Krug

Johannsmeier

Stober

Transfer Learning for Speech Recognition on a Budget. arXiv preprint arXiv:1706.00290.

37.

Spacy. https://spacy.io. Accessed June 19, 2020.

38.

https://www.liveatc.net. Accessed August 21, 2019.

39.

Baevski

Zhou

Mohamed

Auli

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. arXiv preprint arXiv:2006.11477.

40.

Paliwal

K. K.

On the Use of Filter-Bank Energies as Features for Robust Speech Recognition. In ISSPA’99. Proc., 5th International Symposium on Signal Processing and its Applications, Vol. 2. IEEE, 1999, pp. 641–644.

41.

Karita

Chen

Hayashi

Hori

Inaguma

Jiang

Someki

Soplin

N. E. Y.

Yamamoto

Wang

Watanabe

A Comparative Study on Transformer vs RNN in Speech Applications. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 449–456.