Browse State-of-the-Art › Lipreading
Lipreading
36 papers with code · 8 benchmarks · 8 datasets archive 2025-07-28
Lipreading is a process of extracting speech by watching lip movements of a speaker in the absence of sound. Humans lipread all the time without even noticing. It is a big part in communication albeit not as dominant as audio. It is a very helpful skill to learn especially for those who are hard of hearing.
Deep Lipreading is the process of extracting speech from a video of a silent talking face using deep neural networks. It is also known by few other names: Visual Speech Recognition (VSR), Machine Lipreading, Automatic Lipreading etc.
The primary methodology involves two stages: i) Extracting visual and temporal features from a sequence of image frames from a silent talking video ii) Processing the sequence of features into units of speech e.g. characters, words, phrases etc. We can find several implementations of this methodology either done in two separate stages or trained end-to-end in one go.
Description from the archive archive 2025-07-28.
Benchmarks archive 2025-07-28
8 leaderboard tables shown for this task, 8 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.
Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. Each line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the instrument figure counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the row's paper, the line starts with that repository's state (“official” is the archive's flag, not a verdict on who wrote the code; “community repositories only” when every sample that ran came from a community repository, “official: no sample here; runs from other or unrecorded repositories” when some came from a repository the paper names or has in its text, or from none recorded); hover it for the repositories the samples that ran came from. A dash means no Syntology record for that paper, not a recorded non-run. Each “ran” line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Libraries
Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.
Datasets archive 2025-07-28
8 datasets whose archive record lists this task, ordered by the archive's paper count.
Subtasks archive 2025-07-28
1 subtask in the archive's task tree.
Parent tasks archive 2025-07-28
Papers archive 2025-07-28
archive papers tagged: 103 · with a code link: 36 · where Syntology ran a sample: 9 (6 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument) Syntology
Show: all tagged papersonly where code ran (9 of 103 tagged: 6 with a run with no instrument failure, 3 where every run was a failure of Syntology's instrument)
Page 1 of 2: papers 1 to 100 of 103, in archive order: by repositories listed in the archive (most first), then newest first, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers that list no repository come after every paper that lists one.
Papers without a page here are shown as plain text. A Syntology line reads “N ran (of which C constructed an object rather than computing a result; K with no instrument failure: H honoured, V violated, P with no contract checked; I where Syntology's instrument failed) · U unverified”; the figure “where Syntology's instrument failed” counts failures of Syntology's instrument, not of the code. When the archive marks a repository official for the paper, the line starts with that repository's state (the archive's flag, not a verdict on who wrote the code); hover it for the repositories the samples that ran came from. The first sentence of the abstract is shown for the first 30 papers; every abstract is on its paper's page.
-
5 Nov 2016 13 repositories listed Syntology official (archive's flag): 3 ran · 10 ran (of which 0 constructed an object rather than computing a result; 10 with no instrument failure: 0 honoured, 0 violated, 10 with no contract checked; 0 where Syntology's instrument failed) · 5 unverified (of 15 harvested samples)Lipreading is the task of decoding text from the movement of a speaker's mouth.
-
6 Sep 2018 4 repositories listedThe goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
-
12 Mar 2017 4 repositories listed Syntology official: no sample here; runs from other or unrecorded repositories · 2 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 2 where Syntology's instrument failed) · 0 unverified (of 2 harvested samples) · 1 pointer-only (licence)We propose an end-to-end deep learning architecture for word-level visual speech recognition.
-
12 Feb 2021 3 repositories listedIn this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
-
25 Mar 2023 2 repositories listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 0 honoured, 0 violated, 3 with no contract checked; 0 where Syntology's instrument failed) · 3 unverified (of 6 harvested samples) · 2 pointer-only (licence)Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
-
26 Feb 2022 2 repositories listedHowever, these advances are usually due to the larger training sets rather than the model design.
-
5 Jan 2022 2 repositories listedThe lip-reading WER is further reduced to 26.
-
12 May 2020 2 repositories listedVision is often used as a complementary modality for audio speech recognition (ASR), especially in the noisy environment where performance of solo audio modality significantly deteriorates.
-
23 Jan 2020 2 repositories listed Syntology 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples) · 2 pointer-only (licence)We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively.
-
16 Oct 2018 2 repositories listedIt has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers' attributes such as pose, age, gender, and make-up.
-
18 Feb 2018 2 repositories listedIn presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models.
-
9 Feb 2025 1 repository listedWe also introduce a multi-teacher ensemble method to distill the student, which receives audio-visual data as inputs.
-
1 Feb 2025 1 repository listedIn addition, a thorough ablation study is carried out, where it is studied how the different components that form the architecture influence the quality of speech recognition.
-
4 Nov 2024 1 repository listed Syntology official (archive's flag): 3 ran · 3 ran (of which 0 constructed an object rather than computing a result; 3 with no instrument failure: 3 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 6 unverified (of 9 harvested samples) · 9 pointer-only (licence)We demonstrate that training a single model for all three tasks enhances VSR and AVSR performance, overcoming typical optimisation challenges when training from scratch.
-
18 Jun 2024 1 repository listedVisual Speech Recognition (VSR) stands at the intersection of computer vision and speech recognition, aiming to interpret spoken content from visual cues.
-
11 May 2024 1 repository listedIn this paper, we propose Watch Your Mouth, a novel method that leverages depth sensing to enable accurate silent speech recognition.
-
9 May 2024 1 repository listedThe article introduces a novel audio-visual speech command recognition transformer (AVCRFormer) specifically designed for robust AVSR.
-
23 Feb 2024 1 repository listed Syntology official (archive's flag): 10 ran · 10 ran (of which 2 constructed an object rather than computing a result; 6 with no instrument failure: 1 honoured, 0 violated, 5 with no contract checked; 4 where Syntology's instrument failed) · 2 unverified (of 12 harvested samples) · 12 pointer-only (licence)In this paper, we propose a novel framework, namely Visual Speech Processing incorporated with LLMs (VSP-LLM), to maximize the context modeling ability by bringing the overwhelming power of LLMs.
-
12 Feb 2023 1 repository listedSilent speech interface is a promising technology that enables private communications in natural language.
-
12 Dec 2022 1 repository listedWe observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained.
-
20 Sep 2022 1 repository listedThe powerful modeling capabilities of all-attention-based transformer architectures often cause overfitting and - for natural language processing tasks - lead to an implicitly learned internal language model in the…
-
3 Sep 2022 1 repository listedIn this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators.
-
28 Aug 2022 1 repository listedState-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex.
-
4 Apr 2022 1 repository listedWith the multi-head key memories, MVM extracts possible candidate audio features from the memory, which allows the lip reading model to consider the possibility of which pronunciations can be represented from the input…
-
24 Feb 2022 1 repository listed Syntology official (archive's flag): 6 ran · 6 ran (of which 0 constructed an object rather than computing a result; 6 with no instrument failure: 0 honoured, 0 violated, 6 with no contract checked; 0 where Syntology's instrument failed) · 0 unverified (of 6 harvested samples)In particular, audio and visual front-ends are trained on large-scale unimodal datasets, then we integrate components of both front-ends into a larger multimodal framework which learns to recognize parallel audio-visual…
-
5 Jan 2022 1 repository listed Syntology official: no sample here; runs from other or unrecorded repositories · 1 ran (of which 0 constructed an object rather than computing a result; 0 with no instrument failure: 0 honoured, 0 violated, 0 with no contract checked; 1 where Syntology's instrument failed) · 0 unverified (of 1 harvested sample) · 1 pointer-only (licence)Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe.
-
14 Dec 2020 1 repository listed Syntology official (archive's flag): 1 ran · 1 ran (of which 0 constructed an object rather than computing a result; 1 with no instrument failure: 1 honoured, 0 violated, 0 with no contract checked; 0 where Syntology's instrument failed) · 1 unverified (of 2 harvested samples)Extensive experiments show that this simple approach significantly surpasses the state-of-the-art in terms of generalisation to unseen manipulations and robustness to perturbations, as well as shed light on the factors…
-
15 Nov 2020 1 repository listedConsidering the non-negligible effects of these strategies and the existing tough status to train an effective lip reading model, we perform a comprehensive quantitative study and comparative analysis, for the first…
-
13 Jul 2020 1 repository listedHowever, our most promising lightweight models are on par with the current state-of-the-art while showing a reduction of 8.
-
21 May 2020 1 repository listedThe experiments show that our proposed model outperforms various state-of-the-art models and incorporating the memory augmented lateral transformers makes a 3.
-
13 Mar 2020 1 repository listed
-
12 Mar 2020 1 repository listed
-
6 Mar 2020 1 repository listed
-
26 Nov 2019 1 repository listed
-
8 Nov 2019 1 repository listed
-
30 Oct 2017 1 repository listed
-
Learning Speaker-Invariant Visual Features for Lipreading9 Jun 2025 0 repositories listed
-
UniCUE: Unified Recognition and Generation Framework for Chinese Cued Speech Video-to-Speech Generation4 Jun 2025 0 repositories listed
-
OXSeg: Multidimensional attention UNet-based lip segmentation using semi-supervised lip contours8 May 2025 0 repositories listed
-
Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation9 Feb 2025 0 repositories listed
-
RAL:Redundancy-Aware Lipreading Model Based on Differential Learning with Symmetric Views9 Sep 2024 0 repositories listed
-
Enhancing Lip Reading with Multi-Scale Video and Multi-Encoder8 Apr 2024 0 repositories listed
-
Cross-Attention Fusion of Visual and Geometric Features for Large Vocabulary Arabic Lipreading18 Feb 2024 0 repositories listed
-
1 Jan 2024 0 repositories listed
-
Analysis of Visual Features for Continuous Lipreading in Spanish21 Nov 2023 0 repositories listed
-
Investigating the dynamics of hand and lips in French Cued Speech using attention mechanisms and CTC-based decoding14 Jun 2023 0 repositories listed
-
Audio-Visual Speech Enhancement with Score-Based Generative Models2 Jun 2023 0 repositories listed
-
Word-level Persian Lipreading Dataset8 Apr 2023 0 repositories listed
-
17 Feb 2023 0 repositories listed
-
LipFormer: Learning to Lipread Unseen Speakers based on Visual-Landmark Transformers4 Feb 2023 0 repositories listed
-
29 Aug 2022 0 repositories listed
-
Towards MOOCs for Lipreading: Using Synthetic Talking Heads to Train Humans in Lipreading at Scale21 Aug 2022 0 repositories listed
-
Lip-Listening: Mixing Senses to Understand Lips using Cross Modality Knowledge Distillation for Word-Based Models5 Jun 2022 0 repositories listed
-
Is Lip Region-of-Interest Sufficient for Lipreading?28 May 2022 0 repositories listed
-
23 May 2022 0 repositories listed
-
Multistream neural architectures for cued-speech recognition using a pre-trained visual feature extractor and constrained CTC decoding11 Apr 2022 0 repositories listed
-
Self-supervised Transformer for Deepfake Detection2 Mar 2022 0 repositories listed
-
Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition15 Feb 2022 0 repositories listed
-
Personalized One-Shot Lipreading for an ALS Patient2 Nov 2021 0 repositories listed
-
14 Oct 2021 0 repositories listed
-
LRWR: Large-Scale Benchmark for Lip Reading in Russian language14 Sep 2021 0 repositories listed
-
Large-vocabulary Audio-visual Speech Recognition in Noisy Environments10 Sep 2021 0 repositories listed
-
Sign Language Translation in a Healthcare Setting1 Jul 2021 0 repositories listed
-
14 Dec 2020 0 repositories listed
-
FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire6 Aug 2020 0 repositories listed
-
Audio-visual Multi-channel Recognition of Overlapped Speech18 May 2020 0 repositories listed
-
9 Mar 2020 0 repositories listed
-
6 Jan 2020 0 repositories listed
-
28 Nov 2019 0 repositories listed
-
1 Oct 2019 0 repositories listed
-
Alternative Visual Units for an Optimized Phoneme-Based Lipreading System16 Sep 2019 0 repositories listed
-
30 Aug 2019 0 repositories listed
-
14 Aug 2019 0 repositories listed
-
Lipper: Synthesizing Thy Speech using Multi-View Lipreading28 Jun 2019 0 repositories listed
-
LipReading with 3D-2D-CNN BLSTM-HMM and word-CTC models25 Jun 2019 0 repositories listed
-
1 Jun 2019 0 repositories listed
-
Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading4 May 2019 0 repositories listed
-
Learning from Videos with Deep Convolutional LSTM Networks9 Apr 2019 0 repositories listed
-
On the Importance of Video Action Recognition for Visual Lipreading22 Mar 2019 0 repositories listed
-
Pushing the boundaries of audiovisual word recognition using Residual Networks and LSTMs3 Nov 2018 0 repositories listed
-
The speaker-independent lipreading play-off; a survey of lipreading machines24 Oct 2018 0 repositories listed
-
3D Feature Pyramid Attention Module for Robust Visual Speech Recognition15 Oct 2018 0 repositories listed
-
28 Sep 2018 0 repositories listed
-
Visual Speech Language Models14 Sep 2018 0 repositories listed
-
13 Jul 2018 0 repositories listed
-
Can DNNs Learn to Lipread Full Sentences?29 May 2018 0 repositories listed
-
Towards Lipreading Sentences with Active Appearance Models29 May 2018 0 repositories listed
-
Comparing heterogeneous visual gestures for measuring the diversity of visual speech signals8 May 2018 0 repositories listed
-
Comparing phonemes and visemes with DNN-based lipreading8 May 2018 0 repositories listed
-
Visual Speech Enhancement23 Nov 2017 0 repositories listed
-
Decoding visemes: improving machine lipreading3 Oct 2017 0 repositories listed
-
Decoding visemes: improving machine lipreading3 Oct 2017 0 repositories listed
-
Understanding the visual speech signal3 Oct 2017 0 repositories listed
-
Visual gesture variability between talkers in continuous visual speech3 Oct 2017 0 repositories listed
-
Visual speech recognition: aligning terminologies for better understanding3 Oct 2017 0 repositories listed
-
End-to-End Multi-View Lipreading1 Sep 2017 0 repositories listed
-
Improving Speaker-Independent Lipreading with Domain-Adversarial Training4 Aug 2017 0 repositories listed
-
Auxiliary Multimodal LSTM for Audio-visual Speech Recognition and Lipreading16 Jan 2017 0 repositories listed
-
16 Nov 2016 0 repositories listed
-
Lipreading with Long Short-Term Memory29 Jan 2016 0 repositories listed
Syntology lines on 9 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced; each line links to that paper's sample list. Syntology's record for this page last changed ; when this build read Syntology's graph is in the build record. For agents: get_harvested_code_for_paper(arxiv_id) lists each paper's samples; how to connect.
Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections