Browse State-of-the-Art › Speech-to-Text Translation
Speech-to-Text Translation
64 papers with code · 11 benchmarks · 4 datasets archive 2025-07-28
Translate audio signals of speech in one language into text in a foreign language, either in an end-to-end or cascade manner.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
11 leaderboard tables shown for this task, 11 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted. 10 shown of 11 until expanded.
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
4 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
1 subtask in the archive's task tree.
Papers archive 2025-07-28
archive papers tagged: 146 · with a code link: 64 · where Syntology ran a sample: 15 (8 with a run with no instrument failure, 7 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (15 of 146 tagged: 8 with a run with no instrument failure, 7 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 146, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
11 Oct 2020 5 repositories listedWe introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation.
-
22 Aug 2023 4 repositories listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages?
-
7 Oct 2023 2 repositories listed Syntology 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples)Previous mainstream audio-and-text LLMs use discrete audio tokens to represent both input and output audio; however, they suffer from performance degradation on tasks such as automatic speech recognition, speech-to-text…
-
20 May 2022 2 repositories listedPaddleSpeech is an open-source all-in-one speech toolkit.
-
18 Mar 2022 2 repositories listed Syntology 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 5 unverified (of 7 harvested samples)Recently, speech representation learning has improved many speech-related tasks such as speech recognition, speech classification, and speech-to-text translation.
-
9 Feb 2022 2 repositories listedSpeech translation datasets provide manual segmentations of the audios, which are not available in real-world scenarios, and existing segmentation methods usually significantly reduce translation quality at inference…
-
2 Jun 2021 2 repositories listedAdapter modules were recently introduced as an efficient alternative to fine-tuning in NLP.
-
7 May 2021 2 repositories listed Syntology official (archive's flag): 7 ran · 7 ran (of which 0 constructed an object rather than computing a result; 7 with no instrument failure: 0 honoured, 0 violated, 7 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples) · 7 pointer-only (licence)By projecting audio and text features to a common semantic representation, Chimera unifies MT and ST tasks and boosts the performance on ST benchmarks, MuST-C and Augmented Librispeech, to a new state-of-the-art.
-
20 Jul 2020 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)Speech translation has recently become an increasingly popular topic of research, partly due to the development of benchmark datasets.
-
29 May 2025 1 repository listedIn particular, on the English→German task, the system achieves a BLEU of 24.
-
24 May 2025 1 repository listedRecent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced the naturalness and flexibility of human computer interaction by enabling seamless understanding across text, vision, and audio…
-
25 Apr 2025 1 repository listedThe model was fine-tuned on MediBeng, a synthetic dataset created to simulate the types of bilingual interactions often found in healthcare environments.
-
13 Feb 2025 1 repository listedWith the growing influence of Large Language Models (LLMs), there is increasing interest in integrating speech representations with them to enable more seamless multi-modal processing and speech understanding.
-
10 Jan 2025 1 repository listedWhile recent multilingual automatic speech recognition models claim to support thousands of languages, ASR for low-resource languages remains highly unreliable due to limited bimodal speech and text training data.
-
22 Jul 2024 1 repository listedWe introduces LLaST, a framework for building high-performance Large Language model based Speech-to-text Translation systems.
-
19 Jul 2024 1 repository listedWe reveal that the inclusion of code-switching units results in higher translation performance than monolingual settings and that models are better at code-switching translation into English than non-English.
-
9 Jul 2024 1 repository listedSpeech Integrated Large Language Models (SILLMs) combine large language models with speech perception to perform diverse tasks, such as emotion recognition to speaker verification, demonstrating universal audio…
-
27 Jun 2024 1 repository listedRecent efforts to develop NLP technologies for African languages have focused on their standard dialects, resulting in disparities for dialects and varieties for which there are little to no resources or tools.
-
26 Jun 2024 1 repository listedOur models and code are available as open-source resources.
-
20 Jun 2024 1 repository listedThis paper describes the FBK's participation in the Simultaneous Translation Evaluation Campaign at IWSLT 2024.
-
10 Jun 2024 1 repository listedTo fill this gap, we introduce StreamAtt, the first StreamST policy, and propose StreamLAAL, the first StreamST latency metric designed to be comparable with existing metrics for SimulST.
-
5 Jun 2024 1 repository listed Syntology official (archive's flag): 4 ran · 4 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 4 harvested samples)Simultaneous speech-to-speech translation (Simul-S2ST, a. k. a streaming speech translation) outputs target speech while receiving streaming speech inputs, which is critical for real-time communication.
-
18 May 2024 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 2 unverified (of 4 harvested samples)A promising approach to preserving model performance in linearized transformers is to employ position-based re-weighting functions.
-
16 Feb 2024 1 repository listedThe speech encoder seamlessly integrates with the MT model at inference, enabling direct translation from speech to text, across all languages supported by the MT model.
-
30 Dec 2023 1 repository listedThis work evaluated several cutting-edge large-scale foundation models based on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2, and Whisper-large-v3, on three code-switched corpora.
-
1 Nov 2023 1 repository listedConventional speech-to-text translation (ST) systems are trained on single-speaker utterances, and they may not generalize to real-life scenarios where the audio contains conversations by multiple speakers.
-
28 Aug 2023 1 repository listedConsistency regularization methods, such as R-Drop (Liang et al., 2021) and CrossConST (Gao et al., 2023), have achieved impressive supervised and zero-shot performance in the neural machine translation (NMT) field.
-
22 Aug 2023 1 repository listedOur single text encoder, covering 200 languages, substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks.
-
24 May 2023 1 repository listedJoint speech-language training is challenging due to the large demand for training data and GPU consumption, as well as the modality gap between speech and language.
-
19 May 2023 1 repository listedThe key point is to bridge the modality gap between speech and text so that useful MT techniques can be applied to ST.
-
15 May 2023 1 repository listed
-
10 Apr 2023 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 1 pointer-only (licence)
-
1 Mar 2023 1 repository listed
-
27 Jan 2023 1 repository listed
-
19 Dec 2022 1 repository listed
-
28 Oct 2022 1 repository listed
-
24 Oct 2022 1 repository listed
-
3 Jul 2022 1 repository listed
-
9 Jun 2022 1 repository listed Syntology official: harvested, nothing ran · 0 ran · 1 unverified (of 1 harvested sample)
-
5 May 2022 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
2 May 2022 1 repository listed
-
22 Apr 2022 1 repository listed
-
20 Mar 2022 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
11 Jan 2022 1 repository listed
-
21 Dec 2021 1 repository listed
-
15 Sep 2021 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
9 Sep 2021 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 3 unverified (of 5 harvested samples) · 4 pointer-only (licence)
-
7 Sep 2021 1 repository listed
-
11 May 2021 1 repository listed
-
10 May 2021 1 repository listed
-
21 Apr 2021 1 repository listed
-
18 Dec 2020 1 repository listed
-
2 Nov 2020 1 repository listed
-
21 Sep 2020 1 repository listed
-
21 Sep 2020 1 repository listed
-
5 Aug 2020 1 repository listed
-
4 Feb 2020 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 3 harvested samples) · 3 pointer-only (licence)
-
18 Jan 2020 1 repository listed Syntology official (archive's flag): 5 ran · 5 ran (of which 0 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 0 where Syntology's instrument failed) · 6 unverified (of 11 harvested samples)
-
16 Dec 2019 1 repository listed
-
12 Apr 2019 1 repository listed
-
5 Sep 2018 1 repository listed
-
12 Feb 2018 1 repository listed
-
9 Feb 2018 1 repository listed
-
6 Dec 2016 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 5 unverified (of 15 harvested samples)
-
End-to-End Speech Translation for Low-Resource Languages Using Weakly Labeled Data19 Jun 2025 0 repositories listed
-
S2ST-Omni: An Efficient and Scalable Multilingual Speech-to-Speech Translation Framework via Seamless Speech-Text Alignment and Streaming Speech Generation11 Jun 2025 0 repositories listed
-
Improving Language and Modality Transfer in Translation by Character-level Modeling30 May 2025 0 repositories listed
-
Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarios30 May 2025 0 repositories listed
-
AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation18 Mar 2025 0 repositories listed
-
Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision26 Feb 2025 0 repositories listed
-
Balancing Speech Understanding and Generation Using Continual Pre-training for Codec-based Speech LLM24 Feb 2025 0 repositories listed
-
Speech to Speech Translation with Translatotron: A State of the Art Review9 Feb 2025 0 repositories listed
-
When End-to-End is Overkill: Rethinking Cascaded Speech-to-Text Translation1 Feb 2025 0 repositories listed
-
How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?24 Dec 2024 0 repositories listed
-
Representation Purification for End-to-End Speech Translation5 Dec 2024 0 repositories listed
-
Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages11 Nov 2024 0 repositories listed
-
Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?31 Oct 2024 0 repositories listed
-
A Survey on Speech Large Language Models24 Oct 2024 0 repositories listed
-
Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model24 Oct 2024 0 repositories listed
-
Unveiling the Role of Pretraining in Direct Speech Translation26 Sep 2024 0 repositories listed
-
Finetuning End-to-End Models for Estonian Conversational Spoken Language Translation4 Jul 2024 0 repositories listed
-
Investigating Decoder-only Large Language Models for Speech-to-text Translation3 Jul 2024 0 repositories listed
-
NAIST Simultaneous Speech Translation System for IWSLT 202430 Jun 2024 0 repositories listed
-
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data?11 Jun 2024 0 repositories listed
-
Robust Semantic Communications for Speech Transmission8 Mar 2024 0 repositories listed
-
Compact Speech Translation Models via Discrete Speech Units Pretraining29 Feb 2024 0 repositories listed
-
Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?19 Feb 2024 0 repositories listed
-
Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases1 Feb 2024 0 repositories listed
-
Efficient Monotonic Multihead Attention7 Dec 2023 0 repositories listed
-
End-to-End Speech-to-Text Translation: A Survey2 Dec 2023 0 repositories listed
-
COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning3 Nov 2023 0 repositories listed
-
Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approach6 Oct 2023 0 repositories listed
-
Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer5 Oct 2023 0 repositories listed
-
Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing27 Sep 2023 0 repositories listed
-
SpeechAlign: a Framework for Speech Translation Alignment Evaluation20 Sep 2023 0 repositories listed
-
CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders14 Sep 2023 0 repositories listed
-
On decoder-only architecture for speech-to-text and large language model integration8 Jul 2023 0 repositories listed
-
AudioPaLM: A Large Language Model That Can Speak and Listen22 Jun 2023 0 repositories listed
-
Recent Advances in Direct Speech-to-text Translation20 Jun 2023 0 repositories listed
-
Improved Cross-Lingual Transfer Learning For Automatic Speech Translation1 Jun 2023 0 repositories listed
Syntology lines on 16 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections