Visual Speech Recognition

Benchmarks archive 2025-07-28

2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

6 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 62 papers with code (182 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Deep Audio-Visual Speech Recognition
  6 Sep 2018 4 repositories listed
  The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
• Combining Residual Networks with LSTMs for Lipreading
  12 Mar 2017 4 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 1 pointer-only (licence)
  We propose an end-to-end deep learning architecture for word-level visual speech recognition.
• End-to-end Audio-visual Speech Recognition with Conformers
  12 Feb 2021 3 repositories listed
  In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
• The NPU-ASLP-LiAuto System Description for Visual Speech Recognition in CNVSRC 2023
  7 Jan 2024 2 repositories listed
  This paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP-LiAuto (Team 237) in the first Chinese Continuous Visual Speech Recognition Challenge (CNVSRC) 2023, engaging in the fixed and…
• Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
  25 Mar 2023 2 repositories listed Syntology ran 0 of 6 samples · 6 unverified
  Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
• MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition
  9 Mar 2023 2 repositories listed Syntology ran 7 of 7 samples · 0 unverified
  However, despite researchers exploring cross-lingual translation techniques such as machine translation and audio speech translation to overcome language barriers, there is still a shortage of cross-lingual studies on…
• Visual Speech Recognition for Multiple Languages in the Wild
  26 Feb 2022 2 repositories listed
  However, these advances are usually due to the larger training sets rather than the model design.
• LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild
  16 Oct 2018 2 repositories listed
  It has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers' attributes such as pose, age, gender, and make-up.
• Transfer Learning from Visual Speech Recognition to Mouthing Recognition in German Sign Language
  20 May 2025 1 repository listed
  Sign Language Recognition (SLR) systems primarily focus on manual gestures, but non-manual features such as mouth movements, specifically mouthing, provide valuable linguistic information.
• CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization
  6 May 2025 1 repository listed
  They also enable strong performance in Visual Speech Recognition (VSR) with a WER of 20.
• MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens
  14 Mar 2025 1 repository listed
  Audio-Visual Speech Recognition (AVSR) achieves robust speech recognition in noisy environments by combining auditory and visual information.
• Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations
  8 Mar 2025 1 repository listed
  To capture the wide spectrum of phonetic and linguistic diversity, we also introduce a Multilingual Audio-Visual Romanized Corpus (MARC) consisting of 2, 916 hours of audio-visual speech data across 82 languages, along…
• Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models
  9 Feb 2025 1 repository listed
  We also introduce a multi-teacher ensemble method to distill the student, which receives audio-visual data as inputs.
• mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition
  3 Feb 2025 1 repository listed
  In this work, we propose mWhisper-Flamingo for multilingual AVSR which combines the strengths of a pre-trained audio model (Whisper) and video model (AV-HuBERT).
• Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions
  1 Feb 2025 1 repository listed
  In addition, a thorough ablation study is carried out, where it is studied how the different components that form the architecture influence the quality of speech recognition.
• Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation
  23 Jan 2025 1 repository listed Syntology ran 1 of 1 samples · 0 unverified
  Specifically, we suggest a unimodal multi-task learning, which distills cross-modal knowledge and aligns the corrupted modalities, by predicting clean audio targets with corrupted videos, and clean video targets with…
• Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech Recognition
  3 Jan 2025 1 repository listed
  Unlike traditional Automatic Speech Recognition (ASR), Audio-Visual Speech Recognition (AVSR) takes audio and visual signals simultaneously to infer the transcription.
• AlignVSR: Audio-Visual Cross-Modal Alignment for Visual Speech Recognition
  21 Oct 2024 1 repository listed
  Then, based on the temporal correspondence between audio and video, a frame-level local alignment loss is introduced to refine the global alignment, improving the utility of the audio information.
• Large Language Models are Strong Audio-Visual Speech Recognition Learners
  18 Sep 2024 1 repository listed Syntology ran 9 of 12 samples · 3 unverified · 12 pointer-only (licence)
  For example, in the audio and speech domains, an LLM can be equipped with (automatic) speech recognition (ASR) abilities by just concatenating the audio tokens, computed with an audio encoder, and the text tokens to…
• The NPU-ASLP System Description for Visual Speech Recognition in CNVSRC 2024
  5 Aug 2024 1 repository listed
  This paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP (Team 237) in the second Chinese Continuous Visual Speech Recognition Challenge (CNVSRC 2024), engaging in all four tracks,…
• SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data
  1 Aug 2024 1 repository listed
  In this work, we present SynesLM, an unified model which can perform three multimodal language understanding tasks: audio-visual automatic speech recognition(AV-ASR) and visual-aided speech/machine translation(VST/VMT).
• Tailored Design of Audio-Visual Speech Recognition Models using Branchformers
  9 Jul 2024 1 repository listed
  Recent advances in Audio-Visual Speech Recognition (AVSR) have led to unprecedented achievements in the field, improving the robustness of this type of system in adverse, noisy environments.
• Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition
  4 Jul 2024 1 repository listed Syntology ran 10 of 13 samples · 3 unverified
  Cross-modal attention modules are introduced to enrich video features with audio information so that speech variability can be taken into account when training on the video temporal dynamics.
• SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization
  18 Jun 2024 1 repository listed
  Visual Speech Recognition (VSR) stands at the intersection of computer vision and speech recognition, aiming to interpret spoken content from visual cues.
• Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translation
  14 Jun 2024 1 repository listed Syntology ran 5 of 18 samples · 13 unverified · 18 pointer-only (licence)
  Since videos are harder to obtain than audio, the video training data of AVSR models is usually limited to a few thousand hours.
• Watch Your Mouth: Silent Speech Recognition with Depth Sensing
  11 May 2024 1 repository listed
  In this paper, we propose Watch Your Mouth, a novel method that leverages depth sensing to enable accurate silent speech recognition.
• Audio-Visual Speech Recognition based on Regulated Transformer and Spatio-Temporal Fusion Strategy for Driver Assistive Systems
  9 May 2024 1 repository listed
  The article introduces a novel audio-visual speech command recognition transformer (AVCRFormer) specifically designed for robust AVSR.
• A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition
  7 Mar 2024 1 repository listed
  In this paper, we investigate this contrasting phenomenon from the perspective of modality bias and reveal that an excessive modality bias on the audio caused by dropout is the underlying reason.
• Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing
  23 Feb 2024 1 repository listed Syntology ran 8 of 12 samples · 4 unverified · 12 pointer-only (licence)
  In this paper, we propose a novel framework, namely Visual Speech Processing incorporated with LLMs (VSP-LLM), to maximize the context modeling ability by bringing the overwhelming power of LLMs.
• It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition
  8 Feb 2024 1 repository listed Syntology ran 2 of 6 samples · 4 unverified
  Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output.

Syntology lines on 9 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.