Lip Reading

Description from the archive archive 2025-07-28.

Benchmarks archive 2025-07-28

3 leaderboard tables shown for this task, 3 with rows (a “benchmark” on this site is a table with at least one row, as on /sota), ordered by row count. “Best model” is the first row in the archive's own order at snapshot; nothing is re-ranked here and metric direction is not recorded in the archive. PwC's Trend sparklines are not in the archive, so that column is omitted.

Syntology column: samples harvested from the paper's repositories and executed on synthesized fixtures; “ran” is not a correctness claim and does not order the table. A dash means no Syntology record for that paper, not a recorded non-run. Read from the graph 2026-09-24.

Libraries

Not in the archive: the export carries no per-task library table, so there is nothing to show at snapshot 2025-07-28.

Datasets archive 2025-07-28

6 datasets whose archive record lists this task, ordered by the archive's paper count.

Subtasks archive 2025-07-28

1 subtask in the archive's task tree.

Most implemented papers archive 2025-07-28

30 shown of 49 papers with code (153 tagged with this task in all), ordered by repositories listed in the archive, not by stars (the archive holds no stars, so PwC's “Social” and “Latest” sorts cannot be reproduced). Papers without a page here are shown as plain text.

• Deep Audio-Visual Speech Recognition
  6 Sep 2018 4 repositories listed
  The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio.
• Combining Residual Networks with LSTMs for Lipreading
  12 Mar 2017 4 repositories listed Syntology ran 2 of 2 samples · 0 unverified · 1 pointer-only (licence)
  We propose an end-to-end deep learning architecture for word-level visual speech recognition.
• End-to-end Audio-visual Speech Recognition with Conformers
  12 Feb 2021 3 repositories listed
  In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner.
• MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition
  9 Mar 2023 2 repositories listed Syntology ran 7 of 7 samples · 0 unverified
  However, despite researchers exploring cross-lingual translation techniques such as machine translation and audio speech translation to overcome language barriers, there is still a shortage of cross-lingual studies on…
• Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
  5 Jan 2022 2 repositories listed
  The lip-reading WER is further reduced to 26.
• AuthNet: A Deep Learning based Authentication Mechanism using Temporal Facial Feature Movements
  4 Dec 2020 2 repositories listed
  Biometric systems based on Machine learning and Deep learning are being extensively used as authentication mechanisms in resource-constrained environments like smartphones and other small computing devices.
• Lipreading using Temporal Convolutional Networks
  23 Jan 2020 2 repositories listed Syntology ran 1 of 2 samples · 1 unverified · 2 pointer-only (licence)
  We present results on the largest publicly-available datasets for isolated word recognition in English and Mandarin, LRW and LRW1000, respectively.
• LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild
  16 Oct 2018 2 repositories listed
  It has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers' attributes such as pose, age, gender, and make-up.
• Transforming faces into video stories -- VideoFace2.0
  4 May 2025 1 repository listed
  Conducted experiments confirm applicability of the proposed face ReID algorithm that is combining the concepts of face detection, face recognition and passive tracking-by-detection in order to achieve robust and…
• Lend a Hand: Semi Training-Free Cued Speech Recognition via MLLM-Driven Hand Modeling for Barrier-free Communication
  11 Mar 2025 1 repository listed
  Automatic CS Recognition (ACSR) refers to the AI-driven process of automatically recognizing hand gestures and lip movements in CS, converting them into text.
• Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language
  2 Sep 2024 1 repository listed
  To address this challenge, speaker adaptive lip reading technologies have advanced by focusing on effectively adapting a lip reading model to target speakers in the visual modality.
• Audio-Visual Speech Recognition based on Regulated Transformer and Spatio-Temporal Fusion Strategy for Driver Assistive Systems
  9 May 2024 1 repository listed
  The article introduces a novel audio-visual speech command recognition transformer (AVCRFormer) specifically designed for robust AVSR.
• MTGA: Multi-View Temporal Granularity Aligned Aggregation for Event-Based Lip-Reading
  18 Apr 2024 1 repository listed
  Then we design a spatio-temporal fusion module based on temporal granularity alignment, where the global spatial features extracted from event frames, together with the local relative spatial and temporal features…
• Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing
  23 Feb 2024 1 repository listed Syntology ran 8 of 12 samples · 4 unverified · 12 pointer-only (licence)
  In this paper, we propose a novel framework, namely Visual Speech Processing incorporated with LLMs (VSP-LLM), to maximize the context modeling ability by bringing the overwhelming power of LLMs.
• Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism
  11 Dec 2023 1 repository listed
  Our method, which we call NEUral Text to ARticulate Talk (NEUTART), is a talking face generator that uses a joint audiovisual feature space, as well as speech-informed 3D facial reconstructions and a lip-reading loss…
• Do VSR Models Generalize Beyond LRS3?
  23 Nov 2023 1 repository listed
  The Lip Reading Sentences-3 (LRS3) benchmark has primarily been the focus of intense research in visual speech recognition (VSR) during the last few years.
• Learning Separable Hidden Unit Contributions for Speaker-Adaptive Lip-Reading
  8 Oct 2023 1 repository listed
  For deep layers where both the speaker's features and the speech content features are all expressed well, we introduce the speaker-adaptive features to learn for suppressing the speech content irrelevant noise for…
• SelfTalk: A Self-Supervised Commutative Training Diagram to Comprehend 3D Talking Faces
  19 Jun 2023 1 repository listed Syntology ran 3 of 3 samples · 0 unverified · 3 pointer-only (licence)
  To enhance the visual accuracy of generated lip movement while reducing the dependence on labeled data, we propose a novel framework SelfTalk, by involving self-supervision in a cross-modals network system to learn 3D…
• OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
  10 Jun 2023 1 repository listed
  We demonstrate that OpenSR enables modality transfer from one to any in three different settings (zero-, few- and full-shot), and achieves highly competitive zero-shot performance compared to the existing few-shot and…
• A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus
  5 Jun 2023 1 repository listed
  Cued Speech (CS) is a multi-modal visual coding system combining lip reading with several hand cues at the phonetic level to make the spoken language visible to the hearing impaired.
• LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading
  5 Jun 2023 1 repository listed Syntology ran 5 of 8 samples · 3 unverified
  We then condition a diffusion model on the video and use the extracted text through a classifier-guidance mechanism where a pre-trained ASR serves as the classifier.
• Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
  29 Mar 2023 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  To address the problem, we propose using a lip-reading expert to improve the intelligibility of the generated lip regions by penalizing the incorrect generation results.
• GeneFace: Generalized and High-Fidelity Audio-Driven 3D Talking Face Synthesis
  31 Jan 2023 1 repository listed Syntology ran 3 of 4 samples · 1 unverified
  Generating photo-realistic video portrait with arbitrary speech audio is a crucial problem in film-making and virtual reality.
• OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset
  16 Jan 2023 1 repository listed Syntology ran 2 of 2 samples · 0 unverified · 2 pointer-only (licence)
  Inspired by humans comprehending speech in a multi-modal manner, various audio-visual datasets have been constructed.
• Audio-Visual Efficient Conformer for Robust Speech Recognition
  4 Jan 2023 1 repository listed Syntology ran 0 of 6 samples · 6 unverified
  We improve previous lip reading methods using an Efficient Conformer back-end on top of a ResNet-18 visual front-end and by adding intermediate CTC losses between blocks.
• Lip Sync Matters: A Novel Multimodal Forgery Detector
  7 Nov 2022 1 repository listed
  Deepfake technology has advanced a lot, but it is a double-sided sword for the community.
• Relaxed Attention for Transformer Models
  20 Sep 2022 1 repository listed
  The powerful modeling capabilities of all-attention-based transformer architectures often cause overfitting and - for natural language processing tasks - lead to an implicitly learned internal language model in the…
• Training Strategies for Improved Lip-reading
  3 Sep 2022 1 repository listed
  In this paper, we systematically investigate the performance of state-of-the-art data augmentation approaches, temporal models and other training strategies, like self-distillation and using word boundary indicators.
• Speaker-adaptive Lip Reading with User-dependent Padding
  9 Aug 2022 1 repository listed
  In this paper, to remedy the performance degradation of lip reading model on unseen speakers, we propose a speaker-adaptive lip reading method, namely user-dependent padding.
• Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video
  4 Apr 2022 1 repository listed Syntology ran 1 of 1 samples · 0 unverified · 1 pointer-only (licence)
  By learning the interrelationship through the associative bridge, the proposed bridging framework is able to obtain the target modal representations inside the memory network, even with the source modal input only, and…

Syntology lines on 11 of the papers shown; no Syntology record for the others (a paper without an arXiv id cannot be joined to the graph, and absence from the graph layer is not a recorded non-run). “Ran” means the sample executed on a synthesized fixture, not that the paper's result was reproduced. Read from the graph 2026-09-24.