An Edge-Enabled Low-Latency Cross-Lingual Speech-to-Text Framework for Efficient Human

Abstract

Humanoid robots are being introduced in places where people do not speak the same language and people expect quick, natural responses. In such situations, speech interaction cannot afford noticeable delays. Most of the present speech-to-text systems are mainly maintained with cloud servers, leading to latency problems, reliance on dependable connectivity, and failures on the fly when used in real time. These shortcomings become especially evident when robots are expected to autonomously and continuously interact with human users. To address these limitations, this project proposes a new edge-centric speech-to-text framework tailored specifically for the multilingual humanoid robot domain. Instead of sending audio data to the cloud, this method performs speech processing directly within the robot. This technology includes lightweight neural models for real-time streaming, an onboard mechanism that allows for real-time identification of the target language, and local caching methods for quicker retrieval of repeated or known speech patterns. Combine these and you can get quicker, more trustworthy transcription without burning a hole in network resources. The system reduces communication delays to a great extent, while providing transcribed data in multiple languages due to local handling of speech from the wireless edge network. The time taken in overall response is more than 60% lower than the response time used in cloud-based systems; it has been found in experiments. More critically, the framework does a good job with fluctuating network bandwidth, loss of packets, and background noise. It is concluded that edge-based and multilingual speech-to-text systems will be important for humanoid robots to enhance responsivity and contextuality. Understanding faster results in faster reactions, smoother conversations, and moments of interaction that feel more natural is a major step toward pragmatic and reliable communication between humans and robots in the working world.

Keywords

cross-lingual speech recognition humanoid robots low-latency STT real-time interaction wireless edge networks

Introduction

Humanoid robots are transitioning from laboratory prototypes to practical assistants in households, health care, education, and service industries. For these robots to function as effective collaborators, real-time comprehension of natural spoken language is crucial. Unlike command-based interfaces, human conversation is spontaneous, diverse, and often multilingual^1,2 and even sometimes consists of good level of large-scale framework; for example, mSLAM.³ Large-scale multilingual corpora, e.g., Indic-ST,⁴ have made it possible to train and evaluate speech models in low-resource languages. Designing systems that can process speech efficiently under such conditions presents multiple challenges. The first challenge lies in latency. Typical speech-to-text (STT) frameworks normally work well along with cloud-based processing and computing. Generally, all such cloud services, like Google Speech API and Microsoft Azure, provide strong accuracy but incur latency and network dependency issues^5,6 as they incur significant delays because of network transmission and dependability on stable connectivity. Such delays interrupt conversational flow and degrade interaction quality^7,8 and the supposed intelligence and awareness of any robot or humanoid systems. Another issue is related to usability, as real-world environments are linguistically diverse, and any user can switch languages depending upon need during conversation. Maximum existing models typically need distinct training for large-scale multilingual networks, which are computationally heavy^1,2 for each language, or large-scale multilingual networks, which are computationally heavy, resulting in unsuitability for deployment on high-speed resource-constrained wireless edge devices and networks. Another challenge involves wireless network conditions as Humanoid robots commonly interact through Wi-Fi/LTE or emerging 5G networks, which introduce bandwidth fluctuations⁹ and packet loss issues, which have several demerits and need attention in countering the issues of bandwidth fluctuations tackling packet loss in subnet, and latency issues in interference, which keep on compounding, and last but not the least are related to real-time responsiveness (Fig. 1).

FIG. 1.

Speech processing tech.

Recent advancements in multilingual models, streaming speech recognition architectures, and edge computing open up exciting avenues, but are still scattered. High-throughput models like Whisper and its lightweight adaptations generalize well but are resource intensive; they generalize to more challenging languages and noise levels but are just too resource-heavy for edge use cases. Likewise, studies of low latency inference often center around single language benchmarks in controlled environments, limiting their potential for realistic humanoid usage. To overcome these drawbacks, this paper proposes a cross-lingual STT framework that is suitable for edge deployment. It is developed based on: •

Lightweight neural STT architectures (RNN-T and Conformer based) optimized for streaming inference,

•

Adaptive language identification to adapt to the language to make code-switching from one language to the other easier,

•

Edge-level caching and quantization reduce memory overhead in multiple layers to enable faster retrying of interactions,

•

Wireless optimization strategies with long runtime for the unstable network (Fig. 2).

FIG. 2.

Systematic architectural framework novel edge-enabled architecture for real-time, multilingual STT in humanoid robots. STT, speech-to-text.

Unlike existing cross-lingual STT frameworks that primarily focus on improving recognition accuracy through large-scale models, the proposed approach emphasizes real-time deployability under edge constraints. The novelty of this work lies in its unified integration of adaptive language identification, caching, and quantization within a single pipeline optimized for humanoid interaction, rather than introducing isolated optimizations evaluated in offline settings.

The major contributions of this work are summarized as follows: •

We propose an edge-enabled multilingual STT architecture tailored for humanoid robotic systems, designed to operate under strict latency and resource constraints.

•

An adaptive language identification mechanism is introduced to support real-time multilingual and code-switching speech scenarios.

•

Lightweight caching and quantization strategies are integrated to reduce inference latency and memory footprint without compromising recognition performance.

•

The proposed framework is extensively evaluated under realistic network conditions, including varying bandwidth and noise levels, to demonstrate its practical deployability.

Related Work

Research on multilingual STT for humanoid robots intersects with four active areas: (i) cloud-based STT frameworks, (ii) lightweight low-latency architectures, (iii) multilingual and cross-lingual modeling, and (iv) edge computing for speech processing. In all these fields, partial solutions are offered, but they also reveal limitations that drive this study.

Cloud-based speech-to-text systems

Cloud-Based Speech-to-Text Systems Commercial cloud-based STT services provide strong accuracy in data across various languages with the Google Speech API and Microsoft Azure STT and have been widely adopted in multilingual applications such as health care and education systems. However, they require stable connectivity and remote processing, which leads to latencies that disrupt real-time interaction. Studies like OWLS¹⁰ demonstrated that increasing model parameters and training data improve accuracy in high- and low-resource language models, but these models are unsuitable for time-sensitive applications due to network delay and computational weight. Similarly, fine-tuned variants of Whisper¹¹ perform well with challenging dialect-heavy data (such as Swiss German), but they rely on heavy models, restricting on-device usage.

Low-latency neural STT architectures

Recent developments in sequence-to-sequence-like architectures^12,13 such as RNN-T and streaming ASR models¹⁴ with further improvements achieved through knowledge distillation and model compression techniques¹⁵ allow for streaming inference in a shorter time frame. Recent efforts, such as the SM2¹² model demonstrated, trained with weakly supervised data for multilingual streaming recognition, have reported competitive zero-shot translation performance. However, these techniques also show promising limitations regarding latency, such as performance of open streaming dataset (e.g., random forest network). Although they reduce inference time, most are tested in a single language or under restricted conditions, and cross-lingual adaptability and real-world deployment have not been well investigated.

Cross-lingual and multilingual STT

Cross-lingual and Multilingual STT: The study of multilingual corpora, such as MuAViC dataset,^16,17 which combines ∼1,200 hours of audio-visual speech across nine languages, confirms the potential of combining modalities to enhance recognition in noisy contexts.^15,18 However, humanoid robots cannot rely on stable visual input, necessitating robust audio-only multilingual STT. In addition, language switching presents special challenges. Code-switching challenges^19,20 have been highlighted in prior work^19,20 reported that code-switching is more challenging than previously assumed and models struggle to recognize rapid language boundaries. These limitations highlight the need for lightweight, precise, and adaptable language identification techniques^19,21 for real-time multilingual applications, including zero-resource and adversarial language identification approaches.²²

Edge computing for speech processing

Edge Computing for Speech Processing Edge-based inference has recently gained popularity as a technique to alleviate network congestion and latency. Combining quantization and pruning techniques^23–25 shows that large STT models can be condensed for resource-constrained applications with minimal accuracy loss. Yet many focus on single-language cases or single benchmarks. However, comprehensive evaluation of cross-lingual STT on wireless edge nodes—especially under unstable conditions like packet loss or bandwidth variability—is often lacking. Based on our findings, we propose three key gaps in this literature: Latency remains high in cloud-enabled multilingual STT, regardless of transcription accuracy. Code-switching is under-addressed, with few models capable of smooth, low-latency interlanguage translations. Edge optimization strategies have been explored^26,27 for multilingual STT, including containerized edge deployment frameworks for speech processing,²⁸ which has been limited in the literature by wireless, noisy, and resource-limited environments common in humanoid deployment. To fill this gap, this work proposes a framework that includes the following: Streaming-ready multilingual STT architectures tuned for low-latency; Adaptive language identification for real-time code-switching; Use of edge with caching and quantized edge optimization strategies^26,27 has been explored, including containerized edge deployment frameworks for speech processing to achieve speed, accuracy, and resource efficiency. Real-environment validation with network performance and robustness not reported in previous studies.²⁸ This integration broadens the field in terms of accuracy, real-time responsiveness, and edge deployability, all combined in one unified package for humanoid robots at scale in a multilingual context.

Positioning of the proposed work and identified research gaps

Proposed Work Positioning and Research Gaps identified Recent work on multilingual STT systems can be roughly classified into three directions. The first is cloud-based multilingual STT frameworks that are large models for centralized servers to perform well for transcription of multiple languages. The second subset refers to low-latency streaming ASR architectures (e.g., RNN-T, Conformer-based models) to reduce inference delay but are evaluated in controlled or single-language environments.^1,2 The third group studies edge-optimized ASR solutions, but this can only be done in monolingual settings or with naive deployment assumptions. Although improved multilingual accuracy and model scalability are reported in some recent survey papers,^1,2 they are generally related to training efficiency and possible dataset expansion. The elements involved in practical deployment—such as switching from one language to another in real time, wireless instability, and resource constraints at the edge—are often regarded separately or even completely excluded. Consequently, the current literature does not propose a common methodology for each of these problems in realistic robotic or humanoid interaction contexts. Based on this analysis, the following research gaps are identified:

Absence of cross-lingual STT frameworks evaluated under wireless edge deployment conditions.

Limited support for real-time multilingual code-switching in streaming ASR pipelines.

Lack of systematic latency–accuracy–resource trade-off analysis for humanoid and interactive robotic systems.

The proposed framework directly addresses these gaps by integrating adaptive language identification, caching mechanisms,^29,30 and model quantization into a unified edge-oriented STT pipeline. Unlike prior studies that introduce isolated optimizations, this work focuses on end-to-end deployability, enabling low-latency, multilingual speech recognition under practical network and hardware constraints. Highlighting the need for integrated edge-based multilingual STT systems^7,8

Proposed Methodology and Framework

The proposed framework integrates multilingual STT processing, lightweight neural inference, and wireless edge deployment²⁶ to achieve real-time humanoid interaction. The methodology unfolds in five phases, as illustrated in the flowchart Figure 3.

FIG. 3.

Proposed methodology flowchart.

End-to-end workflow and real-time processing pipeline

Figures 3, 4, and 5 represent the full end-to-end workflow of the proposed edge-based multilingual STT framework. Then the incoming audio stream is buffered continuously, and segments are handled in bite-size frames so that speech can be processed in little time. By applying this framing logic, the system could operate as a stream—a requirement for real-time humanoid interaction. A separate language identification module is also operated in parallel with acoustic feature extraction. It minimizes the overall processing latency by studying the segments of speech in parallel instead of sequentially. After the recognition of the spoken language, it dynamically selects the respective lightweight STT model to be executed by the edge device. This enables faster processing when a cached match exists due to bypassing full inference of the model. Moreover, quantized inference is used to balance between recognition accuracy and computational efficiency on the edge hardware, making the architecture suitable for edge hardware. All time-critical features—including language detection, decoding, and post-processing—are automatically scheduled with real-time precision. This coordinated execution maintains speed in speech understanding and response generation to maintain the pace that can allow seamless interaction between humans and humanoid robots.

FIG. 4.

Proposed methodology process flow.

FIG. 5.

Framework diagram for cross-lingual low-latency STT real-time humanoid interaction. STT, speech-to-text.

Dataset preparation

A multilingual corpus was compiled consisting of English, Hindi, Spanish, and Mandarin speech samples. The dataset includes various accents and dialects to enhance robustness, real-world conditions with background noise and overlapping speech, and augmentation techniques (time stretching, pitch shifting, and noise injection) to facilitate generalization (Table 1).

Table 1.

Summary of datasets used for training and evaluation

Dataset	Hours	Sampling rate	Speakers	Languages	Purpose
Public Dataset A	∼160	16 kHz	∼400	EN, ES	Training
Public Dataset B	∼120	16 kHz	∼200	HI, EN	Evaluation
In-house Dataset	∼80	16 kHz	∼50	EN, HI, ES, ZH	Robustness testing

All audio samples were resampled to 16 kHz and normalized prior to training. The in-house dataset consists of speech recorded in indoor environments using consumer-grade microphones, covering variations in accent, speaking rate, and background noise.

Preprocessing

Incoming audio was processed using the method of Voice Activity Detection (VAD) which eliminates silent intervals. Followed by Filters environmental noise while retaining natural features. At next step, feature extraction is done, which is used to determine Mel-Frequency Cepstral Coefficients (MFCCs), and spectrograms were generated for neural input. At the last step of preprocessing, streaming segmentation is done so that audio streams were divided into small frames, enabling incremental transcription with minimal delay.

Model design and training

Two optimized neural models are being practised in the process: RNN-Transducer (RNN-T) and Conformer-based Transformer architectures, which capture both local and global dependencies. A dynamic language identification module has been integrated, automatically routing speech frames to the appropriate language model. Training has been conducted using transfer learning from large pretrained multilingual STT models along with Compression strategies (pruning, quantization, and knowledge distillation) for edge deployability compression techniques^23,25,29 along with structured pruning and distillation techniques.¹⁶

Edge deployment

Edge nodes hosted the optimized models, significantly reducing cloud dependency. The deployment pipeline involved in the framework so that parallel execution of preprocessing → STT inference → humanoid response generation can be achieved. Caching frequently used phrases for fast response in repeated interactions and adaptive strategies for wireless fluctuations, adjusting inference rates dynamically in the edge deployment.

Evaluation metrics

The proposed system framework has been benchmarked using word error rate (WER) for accuracy of transcription, end-to-end latency³¹ giving delay from speech input to humanoid response, throughput and robustness for system stability under bandwidth variations and cross-lingual adaptability that can provide performance under dynamic language-switching scenarios.^32,33

The selected languages were chosen to reflect realistic human–robot interaction scenarios in multilingual environments. English and Hindi represent widely used languages in domestic and institutional settings, while Spanish and Mandarin were included to evaluate cross-lingual scalability. Noise conditions ranging from 0 dB to 30 dB SNR were selected to simulate realistic acoustic environments encountered by humanoid robots, including quiet indoor spaces, moderate background activity, and highly noisy conditions. Packet-loss rates³⁴ were configured to reflect wireless edge deployments³⁵ subject to intermittent connectivity, particularly in mobile or crowded environments.^34,35 These design choices ensure that the evaluation closely mirrors real-world HRI operating conditions (Fig. 6).

FIG. 6.

Proposed CLLL-STT over WEN for RTHI.

Proposed algorithm: Cross-Lingual Low-Latency STT for humanoid robots

Input: Continuous audio stream A(t) Output: Real-time transcriptions T(t) with humanoid responses R(t) 1. Initialize Edge Node E with lightweight STT models M = {M1…Mn} for n languages 2. While audio stream A(t) is active do 3. Apply Voice Activity Detection (VAD) on A(t) → Speech Segments S 4. For each segment s ∈ S do 5. Preprocess s → Features F (MFCC + Spectrogram) 6. Detect language L = LangID(F) 7. Select model ML from M corresponding to L 8. Perform streaming inference: T(t) ← ML(F) 9. Cache frequent phrases for reuse 10. Send transcription T(t) to humanoid response generator 11. Generate humanoid response R(t) 12. End For 13. End While

Noise simulation and calibration procedure

Injecting real-world background noise samples with controlled signal-to-noise ratios from 0 dB to 30 dB acted as the control to simulate noise conditions. Noise simulated data were generated from natural ambient sounds by injecting real-world noise samples into clean speech samples under an SNR setting to evaluate robustness against adverse acoustic conditions. Noise samples were sourced from publicly available environmental sound archives and included office chatter, fan noise, and ambient room noise. The mixing procedure was also standardized by the method of the RMS-based normalization to achieve uniform SNR values for all test samples. With constant steps, different SNR values were used for this study, ranging from 0 dB to 30 dB, with both a moderate and an elevated noise value. This calibration helped achieve repeatability and facilitated equal comparison between models in the same acoustic environment.

Implementation and reproducibility details

Experiments have been performed within PyTorch deep learning framework. All model training and inference were carried out on a hardware platform that can easily take care of edge development with multicore CPU and GPU acceleration. Audio signals were sampled at 16 kHz, and inference of fixed-size audio segments was executed since the system operates in real-time. Less memory usage and less computational cost were associated with implementing reduced-precision weights for quantized inference. Baseline sample size, learning rate, decoding setting, batch size, and other model parameters were unchanged in all the experiments to ensure fair comparison between variables. The same implementation and configuration were used to conduct baseline experiments and ablation experiments. These details are included for reproducibility and independent verification of the reported results.

Simulations and Discussion

This part studies results of the proposed cross-lingual, low-latency STT framework for humanoid robots. The experiments took place in four languages (English, Hindi, Spanish, and Mandarin) under different network and noise conditions. We contrast the approach with two baselines: cloud STT services and a naive edge-only STT system without language adaptation and caching. Results are reported in terms of accuracy, latency, robustness, and adaptability. The hardware specifications, the dataset configurations, and the network conditions used for their evaluation are reported in Table 2 for the simulation parameters. The edge node was designed to be the representation of a resource-constrained deployment to ensure that any improvement observed was feasible for real humanoid robots. Instead of traditional k-fold cross-validation, repeated scenario-based evaluations were adopted due to the streaming and real-time nature of the proposed system.

Table 2.

Simulation parameters summary

Parameter	Value
Edge node CPU (vCPU)	4 vCPU
Edge node RAM (GB)	8 GB
Model types	RNN-T (streaming), conformer (optimized)
Avg. audio frame (ms)	20 ms
VAD window (ms)	30 ms
Quantization	8-bit dynamic quantization
Network types tested	Wi-Fi 6, 4 G LTE, 5G (emulated)
Dataset languages	English, Hindi, Spanish, Mandarin
Noise SNR range (dB)	0–30 dB
Packet loss range (%)	0%–10%

Table 3 compares the average WER, latency, throughput, and robustness across the three systems. The proposed method consistently outperforms the naive edge system and significantly reduces latency compared with the cloud baseline.

Table 3.

Comparison of average word error rate, latency, throughput, and robustness

System	Avg. WER (%)	End-to-end latency (ms)	Throughput (req/s)	Robustness score(0–1)
Proposed edge STT	10	120	40	0.88
Cloud STT	9.5	350	20	0.72
Edge-naive STT	12	200	30	0.65

Proposed edge STT: WER ≈ 10%, latency ≈ 120 ms, throughput ≈ 40 req/s.

Cloud STT: slightly lower WER (9.5%) but ∼3 × higher latency (350 ms).

Edge-naive: moderate latency (200 ms) but higher error rate (12%).

These results validate that our framework provides precise answers in a timely manner for human–robot interaction. We conducted an ablation study (Table 4) to validate the contributions of individual components. Removing caching increased latency by ∼30 ms and slightly degraded accuracy. Excluding language identification forced the model to rely on a single multilingual network, raising WER to 11.5%. Disabling quantization improved WER marginally but at the cost of >3× memory consumption, reducing edge deployability. These findings emphasize that each component—caching, LangID, and quantization—contributes to the system’s efficiency.

Table 4.

Ablation study

Configuration	Avg. WER (%)	Latency (ms)	Memory (MB)
Full proposed (cache + LangID + quant)	10	120	85
No caching	10.8	155	90
No LangID (single multilingual model)	11.5	140	120
No quantization (full precision)	9.6	180	300

WER, word error rate.

A brief ablation study on the contribution of each component toward the general framework was performed by disabling selectively the three major components mentioned in this proposal—adaptive language identification, caching, and quantization. We treat the full configuration of the system as an exemplary comparison. Notably, we can observe that when stripping out the adaptive lightweight language identification techniques,^21,30 there is an obvious increase in latency in multilingual and code-switching situations. This reduction illustrates the critical role of early language detection in avoiding the model switching overhead and unnecessary inference delays, particularly in real-time user interaction. As usual, when you disable this caching mechanism, it means taking longer for the average inference time for more frequently typed commands. Therefore, caching is a simple way of stopping duplicate computation, and this is particularly valuable when we consider humanoid applications, which tend to use very similar repeat speech. With the absence of caching, it seems that the recognition accuracy has certainly not significantly improved, but responsiveness is also not as fast as it used to be. Elimination of quantization will lead to increased memory usage and thus compute performance loss. This will directly affect latency. And though often this model can be good for obtaining marginal accuracy gains under conditions of full-precision inference, it is ill-suited for edge-based deployment in real-time due to its latency penalties. This observation shows that quantization is necessary to find that balance between accuracy and time efficiency. The ablation results show that the independent contribution of each subsystem is high, and the combination results in a good trade-off between latency, robustness, and resource consumption compared with the other modules. This enhances the design decision of a comprehensive framework for real-time multilingual speech-to-text processing on edge devices.

Figure 7 presents latency vs bandwidth for the three systems. While consistent with prior latency-aware STT studies, cloud STT latency sharply rises in low-bandwidth conditions (up to 450 ms at 1 Mbps); the proposed edge STT maintains sub-220 ms latency even under extreme constraints. At higher bandwidths, our system stabilizes around 118–120 ms, ideal for conversational humanoids. This proves that edge-based inference eliminates the dependency on stable high-throughput networks.

FIG. 7.

End-to-end latency variation with available bandwidth.

Analysis of this simulation is explained as the proposed system consistently maintains lower latency at all bandwidth levels. The small accuracy drop (from 9.5% in cloud to ∼10% WER) is likely due to edge quantization and limited context but acceptable for many real-time interactions. Throughput advantages are clear: edge system can serve more concurrent requests due to reduced transmission overhead. Figure 8 depicts WER as a function of SNR. Both proposed and cloud systems perform comparably at medium-to-high SNR (>15 dB). However, at very low SNR (0–5 dB), cloud STT benefits from larger offline training datasets, showing slightly better accuracy. Still, the proposed edge STT remains within 2% to 3% of cloud WER, validating its robustness for noisy, real-world environments similar to multimodal robustness approaches explored in prior work (Table 5).²⁶

FIG. 8.

Noise robustness: WER variation across different SNR levels.

Table 5.

Word error rate (WER) values corresponding to Fig. 7

Method	Noise level (SNR dB)	Mean WER (%)	Std. deviation (%)
Cloud-based STT API	Clean	6.9	0.4
Edge ASR (no optimization)	Clean	9.8	0.6
Edge ASR + quantization	Clean	8.9	0.5
Edge ASR + caching	Clean	8.7	0.5
Proposed framework	Clean	8.4	0.6
Proposed framework	10 dB	9.6	0.7
Proposed framework	0 Db	11.8	0.9

STT, speech-to-text.

The edge model’s robustness holds well in realistic noise, and there is a trade-off, which has very noisy conditions still Favor the large cloud models with richer data; could consider more noise augmentation or ensemble. Figure 9 shows the latency comparison among the three systems, reinforcing that our edge-based design is ∼66% faster than cloud-based transcription. Figure 10 shows throughput degradation under packet loss (Table 6).

FIG. 9.

Average end-to-end latency comparison.

FIG. 10.

Throughput vs packet loss.

Table 6.

End-to-end latency measurements corresponding to Fig. 8

Method	Deployment mode	Mean latency (ms)	Std. deviation (ms)
Cloud-based STT API	Cloud	420	38
Edge ASR (no optimization)	Edge	215	21
Edge ASR + quantization	Edge	165	18
Edge ASR + caching	Edge	148	16
Proposed framework	Edge	128	14

STT, speech-to-text.

Figure 10 throughput performance under increasing packet loss, which provides edge deployment, shows greater resilience to network instability. Along with this, it also indicates effective caching and local inference reduce reliance on repeated transmissions, which is helpful under packet loss. Dynamic language identification was tested under rapid language-switching scenarios (English–Hindi, Hindi–English, English–Mandarin, Spanish–English). As shown in Figure 11, the system adapts within 95–130 ms, ensuring natural and uninterrupted interaction across languages. This demonstrates the effectiveness of the lightweight LangID module.

FIG. 11.

Language switching latency for proposed framework.

Figure 11 gives the indications of latency incurred during cross-lingual language switching. LangID module works effectively even with short utterances, but switch latency could be improved further for some language pairs. The delay includes buffer for confidence threshold: trade-off between false switches vs delay. Figure 12 illustrates the caching hit-rate across repeated interactions. With only three to four repetitions, the hit rate exceeds 40%, and after 10 interactions it stabilizes near 85%. This indicates that in practical scenarios (e.g., humanoids working in repetitive environments like households or hospitals), the system achieves substantial efficiency gains through caching.

FIG. 12.

Caching HR vs repeated interactions.

Figure 12, Caching hit-rate vs number of repeated interactions, shows caching serves repeated utterances well, which is critical for humanoid robots in routine tasks (e.g., greetings, commands). Removing LangID increases WER because interference between languages degrades the multilingual model’s performance. Full precision improves accuracy slightly but at unacceptable resource cost for edge use.

Results and Key Findings

The proposed edge STT provides significantly lower end-to-end latency (≈120 ms) than that of cloud-based STT (≈350 ms) but maintains comparable WER (Proposed ≈10% vs Cloud ≈9.5%). It proves the system does not sacrifice much precision for a huge latency gain—a critical aspect for humanoid responsiveness. Even when bandwidth is low (≤10 Mbps) the proposed system is much more responsive than cloud-based frameworks, since inference occurs locally (though cloud latency degrades sharply when bandwidth is scarce). The noise robustness³¹ (WER vs SNR) demonstrates that our proposed model is capable of performance as compared with cloud STT at medium-to-high SNRs (≥15 dB). Even with very low SNRs (≤5 dB), cloud models trained on larger corpora still have a slight edge. Throughput under packet loss proves that the proposed edge STT is more resilient—throughput decreases gradually with packet loss (from ∼42 req/s → 32 req/s at 10% loss) compared with cloud STT.

Caching gives an operational benefit—when you are doing a lot of caching, its hit-rate goes up quickly, which reduces latency and computation at inference time. All of these (caching, LangID, quantization) are confirmed by ablation: removing caching increases latency and WER slightly; removing quantization reduces accuracy but increases memory footprint. The study uses ablation to highlight how subsystem performance is interlaced and how each subsystem contributes to overall system performance. We also found that by eliminating the adaptive lightweight language identification techniques^21,30 we obtain higher latency on multilingual transitions and the importance of this module for real-time switching for language. And of course, disabling the caching mechanism causes bigger inference latencies between repeated commands, whereas lack of it has a large increase in the computational burden for the repetitive commands without an effect as much on accuracy. The present results demonstrate that each part is used in a unique way for latency reduction, robustness, and use of the resources (Table 7).

Table 7.

Comparison with baseline

Method	Latency (ms)	WER (%)
Cloud STT	420	6.9
Edge ASR (no opt.)	210	9.8
Proposed	130	8.4

STT, speech-to-text; WER, word error rate.

Computational complexity and resource analysis

Computational complexity was considered in the form of inference latency, memory footprint, and processing throughput rather than the abstract asymptotic notation. Quantization reduced the model size and memory access costs, and caching prevented repetitive inference in the case of familiar utterances. This twofold approach together caused reductions in average latency and variance under sustained workloads, therefore validating the suitability of the framework for real-time edge deployment.

Overall trade-offs and limitations latency vs accuracy

Small decrease in WER (<1%–2%) will be offset by huge improvements in latency. Quantizing or pruning reduces memory and compute, but some edge devices having very low memory, or no hardware acceleration at all, would still struggle (Table 8).

Table 8.

Representative error cases

Scenario	Observed error
Code-switching	Missed language boundary
High noise	Word substitution
Fast speech	Partial deletion

Qualitative analysis shows that most transcription errors occur during rapid code-switching or under extremely low SNR conditions, as discussed in robust speech recognition literature. These cases typically involve partial word deletions or substitutions, indicating limitations in language boundary detection rather than acoustic modeling.

Qualitative error analysis

Qualitative analysis indicates that most transcription failures occur during rapid language switching or under extremely low SNR conditions. In such cases, errors typically manifest as word substitutions or partial deletions rather than complete sentence-level failures. Accent-related errors were also observed for speakers not well represented in the training data. These observations suggest that future improvements should focus on faster language boundary detection and improved robustness to accent variability (Table 9).

Table 9.

Representative transcription error cases observed

Scenario	Observed error type	Likely cause
Rapid code-switching	Missed language boundary	Delayed language identification
High background noise (0 dB)	Word substitution	Masked phonetic cues
Fast speaker rate	Partial word deletion	Limited temporal resolution
Accent variation	Phoneme confusion	Insufficient accent diversity

Conclusion and Future Work

This paper proposed an edge-enabled, cross-lingual-based speech-to-text framework for humanoid robots in multilingual, real-world systems. Moving computation from the cloud server to wireless edge nodes, the system proves that the humanoid assistants can deliver the responsive responses necessary for natural human interaction and user-perceived latency constraints in HRI systems.³⁵ Experimental studies revealed a 60%–70% reduction in end-to-end latency compared with the standard cloud-based process while preserving transcription accuracy within 1% to 2% of cloud baselines. Notably, the framework showed resilience against bandwidth variability, packet loss, network degradation effects studied in and background noise and thereby did not suffer from poor performance and was stable under dynamic conditions. Lightweight language identification techniques facilitated smooth code-switching, and compression techniques, including quantization, pruning, and caching, made such deployment possible even on constrained hardware. Collectively, these results underscore that low-latency multilingual, and resource-light STT, a resource-efficient STT, both could be possible and feasible for humanoid robots (HRAs, thus representing a contribution to the human–robot interaction research community). Despite the significant enhancements of the proposed system, there still remain opportunities for improvement. For example, extending to languages with low-resource, underrepresented language varieties, including those with high dialectal diversity and code switching (which has persistent problems with current work). One even more significant opportunity is to improve robustness to harsh acoustic environments. Generalization at the crowded or reverberant environmental condition would be improved by adding room impulse response augmentation using different ambient noise sources in a real-world context as well as sophisticated noise modeling. Apart from the accuracy, the framework can be enhanced further by means of adaptive model scaling that would enable the system to dynamically change the complexity based on the appropriate battery, compute, and network robustness. Hybrid edge–cloud partnership may also be another area, and edge-based inference is good in both response time and privacy shield, but for a noisy and a long utterance and selectively offloading on cloud may also be good. The last critical things remain important—user-centric evaluations remain exceedingly important. The method in practical settings, which are homes, schools, and hospitals, can provide a deep understanding of the effect of response delay, recognition accuracy, and conversational flow on user satisfaction and trust, the study explains. To this end, the recent outcomes have clearly demonstrated a transformational role of a multilingual speech-to-text at the edge to facilitate humanoid robots from mere reactive mechanisms to active participants in a natural context. This way of addressing the problem can be implemented in an appropriate and secure way in a flexible manner in order to develop an open platform to deploy interactive robots across diverse linguistic, acoustic, and network contexts. One of the key contributions of this work is to demonstrate that low-latency, multilingual STT can be achieved without using big cloud-based models. Alternatively, it can be successfully realized on edge devices themselves. Through articulating system-wide limitations and linguistic diversity, the work contributes to the evolution of more flexible, resilient, sustainable humanoid robots that can work in physical real-world environments. The experimental approach also highlights practical trade-offs between latency, accuracy, and resource use. So methods such as aggressive quantization and caching lessen the burden of computation but introduce a time complexity for effective fine-tuning to retain the quality of the recognition in numerous languages as well as noisier environments. This knowledge is particularly critical because speech-enabled robotic systems are required to perform on constrained hardware. The timely communication is important in the natural human–robot interaction. But from the standpoint within a speech-to-text environment, significant delays beyond a permissible threshold will interrupt turn-taking, reduce users’ interactions, and destroy users’ confidence in the system. So, it is necessary to maintain this timing limit. The framework is in a sensible compromise with parallel execution paths to avoid delays, caching to prevent repeated inference, and quantization to reduce computation. Accordingly, it is ideally suited for continuing to work in mobile humanoid robots in dynamic, resource-constrained settings aligning with emerging trends in edge-based speech processing.

Future Research Directions

Next steps will focus on scaling the framework to better accommodate low-resource and underrepresented languages with limited training data and linguistic tools. Concurrently, the solution will be bolstered with dynamic scaling features that can adapt processing and resource utilization to changing runtime scenarios. The hybrid edge–cloud collaboration solution enables more selective offloading for response while still managing low-latency interactions (for complex or computation-heavy requests). In addition to technical enhancements, we are also planning work to conduct live user studies around long-term deployments in humanoid robots. The goal of this work is to measure interaction quality, user acceptance, and system robustness over the long term of the software implementation, providing us with useful feedback and help to finalize further work to make our framework perform reliably in everyday use.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

Funding Information

This work was supported by the Deanship of Scientific Research, Vice Presidency for Graduate Studies and Scientific Research, King Faisal University, Saudi Arabia [Grant No. KFU261134]. This work was supported by Princess Nourah bint Abdulrahman University Researchers Supporting Project number (PNURSP2026R432), Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia.

Abbreviations Used

References

1. Chen

, Tian

, Peng

, et al. OWLS: Scaling laws for multilingual speech recognition and translation models. In: Proceedings of International Conference on Machine Learning 2025, vol. 267. PMLR, 2025, pp. 9121–9145. Available from: https://proceedings.mlr.press/v267/chen25bj.html

2. Feng

, Zhang

, Liu

, et al. Edge-ASR: Towards low-bit quantization of state-of-the-art ASR models. arXiv Preprint 2025.

3. Song

, Zhuo

, Yang

, et al. LoRA-whisper: Parameter-efficient and extensible multilingual ASR. arXiv Preprint 2024.

4. Xue

, Wang

, Li

, et al. and “A Weakly-Supervised Streaming Multilingual Speech Model with Truly Zero-Shot Capability. In: Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE; 2023.

5. Bapna

, Cherry

, Zhang

, et al. mSLAM: Massively multilingual joint pre-training for speech and text. arXiv Preprint 2022.

6. Gow-Smith

, Berard

, Boito

, et al. Multilingual speech translation systems for the IWSLT 2023 low-resource track. arXiv Preprint 2023.

7. Sethiya

, Nair

, Walia

, et al. Indic-ST: A large-scale multilingual corpus for low-resource Speech-to-Text translation. ACM Trans. Asian Low-Resour. Lang. Inf. Process 2025;24(6):Article 60–A25; doi: 10.1145/3736720

8. Liu

, Niehues

. Recent highlights in multilingual and multimodal speech translation. In: Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT), Bangkok, Thailand. IWSLT; 2024; pp. 235–253.

9. Iranzo-Sánchez

, Santamaría-Jordà

, Mas-Mollà

, et al. Speech translation for multilingual medical education leveraged by large language models. Artif Intell Med 2025;166:103147.

10.

10. Kim

, Pinyoanuntapong

, Kim

, et al. Edge vs Cloud: How Do We Balance Cost, Latency, and Quality for Large Language Models Over 5G Networks? In: 2025 IEEE Wireless Communications and Networking Conference (WCNC), Milan, Italy. IEEE; 2025, pp. 1–6.

11.

11. Radford

, Kim

, Xu

, et al. and Robust speech recognition via large-scale weak supervision. In: Proceedings of the 40th International Conference on Machine Learning, vol. 202. PMLR; 2023, pp. 28492–28518. Available from: https://proceedings.mlr.press/v202/radford23a.html

12.

12. Shangguan

, et al. Optimizing speech recognition for the edge (for background edge optimization). ArXiv 2019; doi: 10.48550/arXiv.1909.12408

13.

13. Cui

, Yu

, Jiao

, et al. Recent advances in speech language models: A survey. arXiv Preprint 2024; doi: 10.48550/arXiv.2410.03751

14.

14. Jani

, Panchal

, Patel

, et al. Multilingual Speech Recognition: An In-Depth Review of Applications, Challenges, and Future Directions. In: Communication and Intelligent Systems. ICCIS 2023. Lecture Notes in Networks and Systems, vol 968. ( Sharma

, Shrivastava

, Tripathi

, Wang

, eds) Springer: Singapore: 2024. 10.1007/978-981-97-2079-8_1

15.

15. Shim

, Lee

, Chang

, et al. and Knowledge distillation from non-streaming to streaming ASR encoder using auxiliary non-streaming layer In: Proceedings of the Annual Conference of the International Speech Communication Association. INTERSPEECH; 2023, pp. 1663–1667.

16.

16. Beňo

, Pribiš

, Drahoš

. Edge Container for Speech Recognition. Electronics (Basel) 2021;10(19):2420; doi: 10.3390/electronics10192420

17.

17. He

, Peeters

, et al. Zero-resource language identification under adverse conditions. In: Proceedings of the Annual Conference of the International Speech Communication Association. INTERSPEECH; 2024.

18.

18. Jiang

, Zhang

, Li

, et al. and Accurate and structured pruning for efficient automatic speech recognition. In: Proceedings of the Annual Conference of the International Speech Communication Association. INTERSPEECH; 2023, pp. 4104–4108.

19.

19. BG

, Yadava G

. Automatic speech recognition: Challenges, enhancements, and evaluation metrics. Multimed Tools Appl 2025;84(38):46627–46645; doi: 10.1007/s11042-025-20998-6

20.

20. Bandela

, Sharma Sadhu

, Rathore

, et al. Development of Noise Robust Automatic Speech Recognition System. In: 2023 14th International Conference on Computing Communication and Networking Technologies (ICCCNT). ICCCNT: Delhi, India: 2023, pp. 1–6.

21.

21. Zhang

, et al. Streaming end-to-end multilingual speech recognition with joint language identification. arXiv Preprint 2022.

22.

22. Palivela

, Narvekar

, Asirvatham

, et al. Code-switching ASR for low-resource Indic languages: A Hindi-Marathi case study. IEEE Access 2025;13:9171–9198.

23.

23. Froiz-Míguez

, Fraga-Lamas

, Fernández-CaraméS

. Design, implementation, and practical evaluation of a voice recognition based iot home automation system for low-resource languages and resource-constrained edge iot devices: A system for Galician and mobile opportunistic scenarios. IEEE Access 2023;11:63623–63649.

24.

24. Torkamani

, and, Zarin

. Adaptive edge-cloud inference for speech-to-action systems using ASR and large language models. arXiv Preprint 2025.

25.

25. Wang

, et al. Adaptive neural network quantization for lightweight speaker verification. In: Proceedings of the Annual Conference of the International Speech Communication Association. INTERSPEECH; 2023.

26.

26. Veena

, et al. Multi-Modal Signal Fusion: Enhancing Speech Recognition in Noisy Environments. In: 2023 10th IEEE Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON), Vol. 10. IEEE, 2023.

27.

27. Bataev

. RNN-Transducer-based Losses for Speech Recognition on Noisy Targets. arXiv Preprint 2025.

28.

28. Miao

, Cheng

, Zhang

. Low‐latency transformer model for streaming automatic speech recognition. Electron Lett 2022;58(1):44–46.

29.

29. Mohiuddin

, Fatima

, Khan

, et al. Mobile learning evolution and emerging computing paradigms: An edge-based cloud architecture for reduced latencies and quick response time. Array 2022;16:100259.

30.

30. Prabhavalkar

, Hori

, Sainath

, et al. End-to-end speech recognition: A survey. IEEE/ACM Trans Audio Speech Lang Process 2024;32:325–351.

31.

31. Shao

, Zhou

, Wang

, et al. CleanMel: Mel-spectrogram enhancement for improving both speech quality and ASR. IEEE Trans Audio, Speech Lang Process 2025;33:3202–3214.

32.

32. Alexandridis

, et al. Caching networks: Capitalizing on common speech for ASR. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE; 2022.

33.

33. Kang

, Kim

., and GenPTQ: Green Post-Training Quantization for Large-Scale ASR Models with Mixed-Precision Bit Allocation. In: Findings of the Association for Computational Linguistics: EMNLP 2025. EMNLP; 2025.

34.

34. Yau

, Awiphan

, Bootkrajang

, et al. A Robust Throughput Estimation in Edge-Assisted Adaptive Bitrate Streaming Networks. IEEE Access 2025;13:152598–152607.

35.

35. Kum

, Lee

. Can gestural filler reduce user-perceived latency in conversation with digital humans? Applied Sciences 2022;12(21):10972.

An Edge-Enabled Low-Latency Cross-Lingual Speech-to-Text Framework for Efficient Human–Robot Interaction

Abstract

Keywords

Introduction

Related Work

Cloud-based speech-to-text systems

Low-latency neural STT architectures

Cross-lingual and multilingual STT

Edge computing for speech processing

Positioning of the proposed work and identified research gaps

Proposed Methodology and Framework

End-to-end workflow and real-time processing pipeline

Dataset preparation

Preprocessing

Model design and training

Edge deployment

Evaluation metrics

Proposed algorithm: Cross-Lingual Low-Latency STT for humanoid robots

Noise simulation and calibration procedure

Implementation and reproducibility details

Simulations and Discussion

Results and Key Findings

Computational complexity and resource analysis

Overall trade-offs and limitations latency vs accuracy

Qualitative error analysis

Conclusion and Future Work

Future Research Directions

Footnotes

Author Disclosure Statement

Funding Information

Abbreviations Used

References