Lipreading

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

8 leaderboard tables shown for this task, 8 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

8 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Parent tasks archive 2025-07-28

Most implemented papers archive 2025-07-28

30 shown of 36 papers with code (103 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• LipNet: End-to-End Sentence-level Lipreading
  5 Nov 2016 13 repositories listed Syntology ran 0 of 15 samples · 15 unverified
  Lipreading is the task of decoding text from the movement of a speaker's mouth.
• Deep Audio-Visual Speech Recognition
  6 Sep 2018 4 repositories listed
  The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
• Combining Residual Networks with LSTMs for Lipreading
  12 Mar 2017 4 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 1 pointer-only (licence)
  We propose an end-to-end deep learning architecture for word-level visual speech recognition.
• End-to-end Audio-visual Speech Recognition with Conformers
  12 Feb 2021 3 repositories listed
  In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
• Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
  25 Mar 2023 2 repositories listed Syntology ran 0 of 6 samples · 6 unverified
  Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
• Visual Speech Recognition for Multiple Languages in the Wild
  26 Feb 2022 2 repositories listed
  However, these advances are usually due to the larger training sets rather than the model design.
• Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
  5 Jan 2022 2 repositories listed
  The lip-reading WER is further reduced to 26.
• Discriminative Multi-modality Speech Recognition
  12 May 2020 2 repositories listed
  Vision is often used as a complementary modality for audio speech recognition (ASR), especially in the noisy environment where performance of solo audio modality significantly deteriorates.
• Lipreading using Temporal Convolutional Networks
  23 Jan 2020 2 repositories listed Syntology ran 1 of 2 samples · 1 unverified · 2 pointer-only (licence)
  We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively.
• LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild
  16 Oct 2018 2 repositories listed
  It has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers' attributes such as pose, age, gender, and make-up.
• End-to-end Audiovisual Speech Recognition
  18 Feb 2018 2 repositories listed
  In presence of high levels of noise, the end-to-end audiovisual model significantly outperforms both audio-only models.
• Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models
  9 Feb 2025 1 repository listed
  We also introduce a multi-teacher ensemble method to distill the student, which receives audio-visual data as inputs.
• Evaluation of End-to-End Continuous Spanish Lipreading in Different Data Conditions
  1 Feb 2025 1 repository listed
  In addition, a thorough ablation study is carried out, where it is studied how the different components that form the architecture influence the quality of speech recognition.
• Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs
  4 Nov 2024 1 repository listed Syntology ran 0 of 9 samples · 9 unverified · 9 pointer-only (licence)
  We demonstrate that training a single model for all three tasks enhances VSR and AVSR performance, overcoming typical optimisation challenges when training from scratch.
• SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization
  18 Jun 2024 1 repository listed
  Visual Speech Recognition (VSR) stands at the intersection of computer vision and speech recognition, aiming to interpret spoken content from visual cues.
• Watch Your Mouth: Silent Speech Recognition with Depth Sensing
  11 May 2024 1 repository listed
  In this paper, we propose Watch Your Mouth, a novel method that leverages depth sensing to enable accurate silent speech recognition.
• Audio-Visual Speech Recognition based on Regulated Transformer and Spatio-Temporal Fusion Strategy for Driver Assistive Systems
  9 May 2024 1 repository listed
  The article introduces a novel audio-visual speech command recognition transformer (AVCRFormer) specifically designed for robust AVSR.
• Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing
  23 Feb 2024 1 repository listed Syntology ran 8 of 12 samples · 4 unverified · 12 pointer-only (licence)
  In this paper, we propose a novel framework, namely Visual Speech Processing incorporated with LLMs (VSP-LLM), to maximize the context modeling ability by bringing the overwhelming power of LLMs.
• LipLearner: Customizable Silent Speech Interactions on Mobile Devices
  12 Feb 2023 1 repository listed
  Silent speech interface is a promising technology that enables private communications in natural language.
• Jointly Learning Visual and Auditory Speech Representations from Raw Data
  12 Dec 2022 1 repository listed
  We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained.
• Relaxed Attention for Transformer Models
  20 Sep 2022 1 repository listed
  The powerful modeling capabilities of all-attention-based transformer architectures often cause overfitting and - for natural language processing tasks - lead to an implicitly learned internal language model in the…
• Training Strategies for Improved Lip-reading
  3 Sep 2022 1 repository listed
  In this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators.
• Bayesian Neural Network Language Modeling for Speech Recognition
  28 Aug 2022 1 repository listed
  State-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex.
• Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading
  4 Apr 2022 1 repository listed
  With the multi-head key memories, MVM extracts possible candidate audio features from the memory, which allows the lip reading model to consider the possibility of which pronunciations can be represented from the input…
• Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition
  24 Feb 2022 1 repository listed Syntology ran 0 of 6 samples · 6 unverified
  In particular, audio and visual front-ends are trained on large-scale unimodal datasets, then we integrate components of both front-ends into a larger multimodal framework which learns to recognize parallel audio-visual…
• Robust Self-Supervised Audio-Visual Speech Recognition
  5 Jan 2022 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  Audio-based automatic speech recognition (ASR) degrades significantly in noisy environments and is particularly vulnerable to interfering speech, as the model cannot determine which speaker to transcribe.
• Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
  14 Dec 2020 1 repository listed Syntology ran 1 of 2 samples · 1 unverified
  Extensive experiments show that this simple approach significantly surpasses the state-of-the-art in terms of generalisation to unseen manipulations and robustness to perturbations, as well as shed light on the factors…
• Learn an Effective Lip Reading Model without Pains
  15 Nov 2020 1 repository listed
  Considering the non-negligible effects of these strategies and the existing tough status to train an effective lip reading model, we perform a comprehensive quantitative study and comparative analysis, for the first…
• Towards Practical Lipreading with Distilled and Efficient Models
  13 Jul 2020 1 repository listed
  However, our most promising lightweight models are on par with the current state-of-the-art while showing a reduction of 8.
• SpotFast Networks with Memory Augmented Lateral Transformers for Lipreading
  21 May 2020 1 repository listed
  The experiments show that our proposed model outperforms various state-of-the-art models and incorporating the memory augmented lateral transformers makes a 3.

Syntology lines on 9 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.