Audio-Visual Speech Recognition

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

4 leaderboard tables shown for this task, 4 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

7 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 42 papers with code (100 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Deep Audio-Visual Speech Recognition
  6 Sep 2018 4 repositories listed
  The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
• End-to-end Audio-visual Speech Recognition with Conformers
  12 Feb 2021 3 repositories listed
  In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
• Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
  25 Mar 2023 2 repositories listed Syntology ran 0 of 6 samples · 6 unverified
  Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
• Discriminative Multi-modality Speech Recognition
  12 May 2020 2 repositories listed
  Vision is often used as a complementary modality for audio speech recognition (ASR), especially in the noisy environment where performance of solo audio modality significantly deteriorates.
• CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization
  6 May 2025 1 repository listed
  They also enable strong performance in Visual Speech Recognition (VSR) with a WER of 20.
• MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens
  14 Mar 2025 1 repository listed
  Audio-Visual Speech Recognition (AVSR) achieves robust speech recognition in noisy environments by combining auditory and visual information.
• Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations
  8 Mar 2025 1 repository listed
  To capture the wide spectrum of phonetic and linguistic diversity, we also introduce a Multilingual Audio-Visual Romanized Corpus (MARC) consisting of 2, 916 hours of audio-visual speech data across 82 languages, along…
• Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models
  9 Feb 2025 1 repository listed
  We also introduce a multi-teacher ensemble method to distill the student, which receives audio-visual data as inputs.
• mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition
  3 Feb 2025 1 repository listed
  In this work, we propose mWhisper-Flamingo for multilingual AVSR which combines the strengths of a pre-trained audio model (Whisper) and video model (AV-HuBERT).
• Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation
  23 Jan 2025 1 repository listed Syntology ran 1 of 1 samples · 0 unverified
  Specifically, we suggest a unimodal multi-task learning, which distills cross-modal knowledge and aligns the corrupted modalities, by predicting clean audio targets with corrupted videos, and clean video targets with…
• Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition
  3 Jan 2025 1 repository listed
  Unlike traditional Automatic Speech Recognition (ASR), Audio-Visual Speech Recognition (AVSR) takes audio and visual signals simultaneously to infer the transcription.
• Large Language Models are Strong Audio-Visual Speech Recognition Learners
  18 Sep 2024 1 repository listed Syntology ran 9 of 12 samples · 3 unverified · 12 pointer-only (licence)
  For example, in the audio and speech domains, an LLM can be equipped with (automatic) speech recognition (ASR) abilities by just concatenating the audio tokens, computed with an audio encoder, and the text tokens to…
• SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data
  1 Aug 2024 1 repository listed
  In this work, we present SynesLM, an unified model which can perform three multimodal language understanding tasks: audio-visual automatic speech recognition(AV-ASR) and visual-aided speech/machine translation(VST/VMT).
• Tailored Design of Audio-Visual Speech Recognition Models using Branchformers
  9 Jul 2024 1 repository listed
  Recent advances in Audio-Visual Speech Recognition (AVSR) have led to unprecedented achievements in the field, improving the robustness of this type of system in adverse, noisy environments.
• Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition
  4 Jul 2024 1 repository listed Syntology ran 10 of 13 samples · 3 unverified
  Cross-modal attention modules are introduced to enrich video features with audio information so that speech variability can be taken into account when training on the video temporal dynamics.
• Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation
  14 Jun 2024 1 repository listed Syntology ran 5 of 18 samples · 13 unverified · 18 pointer-only (licence)
  Since videos are harder to obtain than audio, the video training data of AVSR models is usually limited to a few thousand hours.
• Audio-Visual Speech Recognition based on Regulated Transformer and Spatio-Temporal Fusion Strategy for Driver Assistive Systems
  9 May 2024 1 repository listed
  The article introduces a novel audio-visual speech command recognition transformer (AVCRFormer) specifically designed for robust AVSR.
• A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition
  7 Mar 2024 1 repository listed
  In this paper, we investigate this contrasting phenomenon from the perspective of modality bias and reveal that an excessive modality bias on the audio caused by dropout is the underlying reason.
• It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition
  8 Feb 2024 1 repository listed Syntology ran 2 of 6 samples · 4 unverified
  Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output.
• Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation
  7 Jan 2024 1 repository listed Syntology ran 6 of 9 samples · 3 unverified · 9 pointer-only (licence)
  Considering that visual information helps to improve speech recognition performance in noisy scenes, in this work we propose a multichannel multi-modal speech self-supervised learning framework AV-wav2vec2, which…
• RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation
  29 Sep 2023 1 repository listed Syntology ran 0 of 1 samples · 1 unverified
  This is the first time-frequency domain audio-visual speech separation method to outperform all contemporary time-domain counterparts.
• Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder
  14 Aug 2023 1 repository listed
  In this paper, we propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework.
• MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
  18 Jun 2023 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  In this paper, we aim to learn the shared representations across modalities to bridge their gap.
• Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition
  18 Jun 2023 1 repository listed Syntology ran 1 of 2 samples · 1 unverified · 2 pointer-only (licence)
  In this work, we investigate the noise-invariant visual modality to strengthen robustness of AVSR, which can adapt to any testing noises while without dependence on noisy training data, a.
• OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
  10 Jun 2023 1 repository listed
  We demonstrate that OpenSR enables modality transfer from one to any in three different settings (zero-, few- and full-shot), and achieves highly competitive zero-shot performance compared to the existing few-shot and…
• MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information
  4 Jun 2023 1 repository listed
  Audio-visual speech recognition (AVSR) gains increasing attention from researchers as an important part of human-computer interaction.
• Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization
  18 May 2023 1 repository listed
  We investigate the emergent abilities of the recently proposed web-scale speech model Whisper, by adapting it to unseen tasks with prompt engineering.
• Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition
  16 May 2023 1 repository listed
  However, most existing AVSR approaches simply fuse the audio and visual features by concatenation, without explicit interactions to capture the deep correlations between them, which results in sub-optimal multimodal…
• Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring
  15 Mar 2023 1 repository listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models.
• MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
  1 Mar 2023 1 repository listed
  We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages.

Syntology lines on 11 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.