Browse State-of-the-Art › Visual Speech Recognition
Visual Speech Recognition
62 papers with code · 2 benchmarks · 6 datasets archive 2025-07-28
Benchmarks archive 2025-07-28
2 leaderboard tables shown for this task, 2 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
| Dataset | Best model (first row in archive order) | Paper | Code | Syntology | Compare |
|---|---|---|---|---|---|
| LRS3-TED (3 rows) | CTC/Attention | Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels | code | Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified | Compare |
| LRS2 (2 rows) | VTP with more data | Sub-word Level Lip Reading With Visual Attention | — | — | Compare |
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
6 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
1 subtask in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 182 · with a code link: 62 · where Syntology ran a sample: 19 (16 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (19 of 182 tagged: 16 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 182, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
6 Sep 2018 4 repositories listedThe goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
-
12 Mar 2017 4 repositories listed Syntology official: no sample here; runs from other or unrecorded repositories · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 1 pointer-only (licence)We propose an end-to-end deep learning architecture for word-level visual speech recognition.
-
12 Feb 2021 3 repositories listedIn this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
-
7 Jan 2024 2 repositories listedThis paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP-LiAuto (Team 237) in the first Chinese Continuous Visual Speech Recognition Challenge (CNVSRC) 2023, engaging in the fixed and…
-
25 Mar 2023 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples) · 2 pointer-only (licence)Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
-
9 Mar 2023 2 repositories listed Syntology community repositories only · 7 ran (of which 0 constructed an object rather than computing a result; 4 with no instrument failure: 0 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 0 unverified (of 7 harvested samples) · 3 pointer-only (licence)However, despite researchers exploring cross-lingual translation techniques such as machine translation and audio speech translation to overcome language barriers, there is still a shortage of cross-lingual studies on…
-
26 Feb 2022 2 repositories listedHowever, these advances are usually due to the larger training sets rather than the model design.
-
16 Oct 2018 2 repositories listedIt has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers' attributes such as pose, age, gender, and make-up.
-
20 May 2025 1 repository listedSign Language Recognition (SLR) systems primarily focus on manual gestures, but non-manual features such as mouth movements, specifically mouthing, provide valuable linguistic information.
-
6 May 2025 1 repository listedThey also enable strong performance in Visual Speech Recognition (VSR) with a WER of 20.
-
MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens14 Mar 2025 1 repository listedAudio-Visual Speech Recognition (AVSR) achieves robust speech recognition in noisy environments by combining auditory and visual information.
-
8 Mar 2025 1 repository listedTo capture the wide spectrum of phonetic and linguistic diversity, we also introduce a Multilingual Audio-Visual Romanized Corpus (MARC) consisting of 2, 916 hours of audio-visual speech data across 82 languages, along…
-
9 Feb 2025 1 repository listedWe also introduce a multi-teacher ensemble method to distill the student, which receives audio-visual data as inputs.
-
3 Feb 2025 1 repository listedIn this work, we propose mWhisper-Flamingo for multilingual AVSR which combines the strengths of a pre-trained audio model (Whisper) and video model (AV-HuBERT).
-
1 Feb 2025 1 repository listedIn addition, a thorough ablation study is carried out, where it is studied how the different components that form the architecture influence the quality of speech recognition.
-
23 Jan 2025 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample)Specifically, we suggest a unimodal multi-task learning, which distills cross-modal knowledge and aligns the corrupted modalities, by predicting clean audio targets with corrupted videos, and clean video targets with…
-
3 Jan 2025 1 repository listedUnlike traditional Automatic Speech Recognition (ASR), Audio-Visual Speech Recognition (AVSR) takes audio and visual signals simultaneously to infer the transcription.
-
21 Oct 2024 1 repository listedThen, based on the temporal correspondence between audio and video, a frame-level local alignment loss is introduced to refine the global alignment, improving the utility of the audio information.
-
18 Sep 2024 1 repository listed Syntology official (archive's flag): 9 ran · 9 ran (of which 2 constructed an object rather than computing a result; 5 with no instrument failure: 0 honoured, 0 violated, 5 with no contract checked; 4 where Syntology's instrument failed) · 3 unverified (of 12 harvested samples) · 12 pointer-only (licence)For example, in the audio and speech domains, an LLM can be equipped with (automatic) speech recognition (ASR) abilities by just concatenating the audio tokens, computed with an audio encoder, and the text tokens to…
-
5 Aug 2024 1 repository listedThis paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP (Team 237) in the second Chinese Continuous Visual Speech Recognition Challenge (CNVSRC 2024), engaging in all four tracks…
-
1 Aug 2024 1 repository listedIn this work, we present SynesLM, an unified model which can perform three multimodal language understanding tasks: audio-visual automatic speech recognition(AV-ASR) and visual-aided speech/machine translation(VST/VMT).
-
9 Jul 2024 1 repository listedRecent advances in Audio-Visual Speech Recognition (AVSR) have led to unprecedented achievements in the field, improving the robustness of this type of system in adverse, noisy environments.
-
4 Jul 2024 1 repository listed Syntology official (archive's flag): 12 ran · 12 ran (of which 2 constructed an object rather than computing a result; 7 with no instrument failure: 2 honoured, 0 violated, 5 with no contract checked; 5 where Syntology's instrument failed) · 1 unverified (of 13 harvested samples) · 8 pointer-only (licence)Cross-modal attention modules are introduced to enrich video features with audio information so that speech variability can be taken into account when training on the video temporal dynamics.
-
18 Jun 2024 1 repository listedVisual Speech Recognition (VSR) stands at the intersection of computer vision and speech recognition, aiming to interpret spoken content from visual cues.
-
14 Jun 2024 1 repository listed Syntology official (archive's flag): 11 ran · 11 ran (of which 0 constructed an object rather than computing a result; 8 with no instrument failure: 5 honoured, 1 violated, 2 with no contract checked; 3 where Syntology's instrument failed) · 7 unverified (of 18 harvested samples) · 18 pointer-only (licence)Since videos are harder to obtain than audio, the video training data of AVSR models is usually limited to a few thousand hours.
-
11 May 2024 1 repository listedIn this paper, we propose Watch Your Mouth, a novel method that leverages depth sensing to enable accurate silent speech recognition.
-
9 May 2024 1 repository listedThe article introduces a novel audio-visual speech command recognition transformer (AVCRFormer) specifically designed for robust AVSR.
-
7 Mar 2024 1 repository listedIn this paper, we investigate this contrasting phenomenon from the perspective of modality bias and reveal that an excessive modality bias on the audio caused by dropout is the underlying reason.
-
23 Feb 2024 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 2 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 12 harvested samples) · 12 pointer-only (licence)In this paper, we propose a novel framework, namely Visual Speech Processing incorporated with LLMs (VSP-LLM), to maximize the context modeling ability by bringing the overwhelming power of LLMs.
-
8 Feb 2024 1 repository listed Syntology 3 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 1 honoured, 0 violated, 1 with no contract checked; 1 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples) · 1 pointer-only (licence)Recent studies have successfully shown that large language models (LLMs) can be successfully used for generative error correction (GER) on top of the automatic speech recognition (ASR) output.
-
18 Jan 2024 1 repository listed
-
7 Jan 2024 1 repository listed Syntology official (archive's flag): 8 ran · 8 ran (of which 2 constructed an object rather than computing a result; 5 with no instrument failure: 1 honoured, 0 violated, 4 with no contract checked; 3 where Syntology's instrument failed) · 1 unverified (of 9 harvested samples) · 9 pointer-only (licence)
-
23 Nov 2023 1 repository listed
-
21 Nov 2023 1 repository listed
-
15 Sep 2023 1 repository listed
-
14 Aug 2023 1 repository listed
-
18 Jun 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
18 Jun 2023 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 1 constructed an object rather than computing a result; 1 with no instrument failure: 0 honoured, 0 violated, 1 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified; the one sample that ran constructed an object rather than computing a result (of 1 harvested sample) · 1 pointer-only (licence)
-
10 Jun 2023 1 repository listed
-
4 Jun 2023 1 repository listed
-
18 May 2023 1 repository listed
-
16 May 2023 1 repository listed
-
15 Mar 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 1 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
1 Mar 2023 1 repository listed
-
16 Jan 2023 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 2 pointer-only (licence)
-
12 Dec 2022 1 repository listed
-
13 Jul 2022 1 repository listed
-
1 Jun 2022 1 repository listed
-
24 Feb 2022 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)
-
11 Jan 2022 1 repository listed Syntology official (archive's flag): 2 ran · 2 ran (of which 0 constructed an object rather than computing a result; 2 with no instrument failure: 0 honoured, 0 violated, 2 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 3 harvested samples)
-
5 Jan 2022 1 repository listed Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)
-
14 Dec 2020 1 repository listed
-
14 Dec 2020 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)
-
15 Nov 2020 1 repository listed
-
19 May 2020 1 repository listed
-
17 Apr 2020 1 repository listed
-
6 Mar 2020 1 repository listed
-
8 Nov 2019 1 repository listed
-
29 Jan 2019 1 repository listed
-
3 Sep 2018 1 repository listed
-
23 Jul 2018 1 repository listed
-
30 Oct 2017 1 repository listed
-
VisualSpeaker: Visually-Guided 3D Avatar Lip Synthesis8 Jul 2025 0 repositories listed
-
ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition5 Jun 2025 0 repositories listed
-
Cocktail-Party Audio-Visual Speech Recognition2 Jun 2025 0 repositories listed
-
CNVSRC 2024: The Second Chinese Continuous Visual Speech Recognition Challenge27 May 2025 0 repositories listed
-
Leveraging Large Language Models in Visual Speech Recognition: Model Scaling, Context-Aware Decoding, and Iterative Polishing27 May 2025 0 repositories listed
-
Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach20 May 2025 0 repositories listed
-
The Multimodal Information Based Speech Processing (MISP) 2025 Challenge: Audio-Visual Diarization and Recognition20 May 2025 0 repositories listed
-
SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer7 May 2025 0 repositories listed
-
Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides21 Apr 2025 0 repositories listed
-
Visual-Aware Speech Recognition for Noisy Scenarios9 Apr 2025 0 repositories listed
-
Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs9 Mar 2025 0 repositories listed
-
NaturalL2S: End-to-End High-quality Multispeaker Lip-to-Speech Synthesis with Differential Digital Signal Processing17 Feb 2025 0 repositories listed
-
MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition11 Feb 2025 0 repositories listed
-
Lightweight Operations for Visual Speech Recognition7 Feb 2025 0 repositories listed
-
Adapter-Based Multi-Agent AVSR Extension for Pre-Trained ASR Models3 Feb 2025 0 repositories listed
-
LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition8 Jan 2025 0 repositories listed
-
Uncovering the Visual Contribution in Audio-Visual Speech Recognition22 Dec 2024 0 repositories listed
-
Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective29 Sep 2024 0 repositories listed
-
Enhancing CTC-Based Visual Speech Recognition11 Sep 2024 0 repositories listed
-
DCIM-AVSR : Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module31 Aug 2024 0 repositories listed
-
MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization25 Jun 2024 0 repositories listed
-
CNVSRC 2023: The First Chinese Continuous Visual Speech Recognition Challenge14 Jun 2024 0 repositories listed
-
Learn2Talk: 3D Talking Face Learns from 2D Talking Face19 Apr 2024 0 repositories listed
-
XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception21 Mar 2024 0 repositories listed
-
Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer14 Mar 2024 0 repositories listed
-
JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition4 Mar 2024 0 repositories listed
-
Comparison of Conventional Hybrid and CTC/Attention Decoders for Continuous Visual Speech Recognition20 Feb 2024 0 repositories listed
-
SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition18 Jan 2024 0 repositories listed
-
MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition7 Jan 2024 0 repositories listed
-
LiteVSR: Efficient Visual Speech Recognition by Learning from Speech Representations of Unlabeled Data15 Dec 2023 0 repositories listed
-
The GUA-Speech System Description for CNVSRC Challenge 202312 Dec 2023 0 repositories listed
-
Analysis of Visual Features for Continuous Lipreading in Spanish21 Nov 2023 0 repositories listed
-
Speaker-Adapted End-to-End Visual Speech Recognition for Continuous Spanish21 Nov 2023 0 repositories listed
-
End-to-End Lip Reading in Romanian with Cross-Lingual Domain Adaptation and Lateral Inhibition7 Oct 2023 0 repositories listed
-
AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition29 Sep 2023 0 repositories listed
-
The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extraction15 Sep 2023 0 repositories listed
-
20 Aug 2023 0 repositories listed
-
AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model15 Aug 2023 0 repositories listed
Syntology lines on 18 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections