{"url":"/sota/lipreading-on-lip-reading-in-the-wild","task":{"name":"Lipreading","url":"/task/lipreading","note":null},"dataset":{"name":"Lip Reading in the Wild","url":"/dataset/lrw"},"category":"Computer Vision","categories":["Computer Vision"],"category_note":null,"description":"Lipreading is a process of extracting speech by watching lip movements of a speaker in the absence of sound. Humans lipread all the time without even noticing. It is a big part in communication albeit not as dominant as audio. It is a very helpful skill to learn especially for those who are hard of hearing. \r\n\r\nDeep Lipreading is the process of extracting speech from a video of a silent talking face using deep neural networks.  It is also known by few other names: Visual Speech Recognition (VSR), Machine Lipreading, Automatic Lipreading etc. \r\n\r\nThe primary methodology involves two stages: i) Extracting visual and temporal features from a sequence of image frames from a silent talking video ii) Processing the sequence of features into units of speech e.g. characters, words, phrases etc. We can find several implementations of this methodology either done in two separate stages or trained end-to-end in one go.","description_from":"task","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","rank":"the archive's row order at snapshot; not re-ranked","rows_end_at":"2025-07-28","rows_withheld_as_spam":0,"metric_values":"the archive's strings, untouched"},"metrics":["Top-1 Accuracy"],"metric_direction":{"note":"inferred from the metric name only (the archive records no direction); null = not inferred, chart draws points only","by_metric":{"Top-1 Accuracy":"higher"}},"counts":{"rows":22,"rows_with_code":18,"rows_with_paper_page":22,"rows_dated":22,"rows_using_additional_data":1},"rows":[{"rank_in_archive_order":1,"model":"SyncVSR (Word Boundary)","metrics":{"Top-1 Accuracy":"95.0"},"uses_additional_data":false,"paper_date":"2024-06-18","paper":"/paper/syncvsr-data-efficient-visual-speech","paper_url":"https://arxiv.org/abs/2406.12233v1","paper_title":"SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization","code":"https://github.com/KAIST-AILab/SyncVSR","n_code_links":1,"syntology":null},{"rank_in_archive_order":2,"model":"3D Conv + ResNet-18 + DC-TCN + KD (Ensemble & Word Boundary)","metrics":{"Top-1 Accuracy":"94.1"},"uses_additional_data":true,"paper_date":"2022-09-03","paper":"/paper/training-strategies-for-improved-lip-reading","paper_url":"https://arxiv.org/abs/2209.01383v3","paper_title":"Training Strategies for Improved Lip-reading","code":"https://github.com/mpc001/Lipreading_using_Temporal_Convolutional_Networks","n_code_links":1,"syntology":null},{"rank_in_archive_order":3,"model":"SyncVSR","metrics":{"Top-1 Accuracy":"93.2"},"uses_additional_data":false,"paper_date":"2024-06-18","paper":"/paper/syncvsr-data-efficient-visual-speech","paper_url":"https://arxiv.org/abs/2406.12233v1","paper_title":"SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization","code":"https://github.com/KAIST-AILab/SyncVSR","n_code_links":1,"syntology":null},{"rank_in_archive_order":4,"model":"AVCRFormer","metrics":{"Top-1 Accuracy":"89.57"},"uses_additional_data":false,"paper_date":"2024-05-09","paper":"/paper/audio-visual-speech-recognition-based-on","paper_url":"https://www.sciencedirect.com/science/article/pii/S095741742401025X","paper_title":"Audio-Visual Speech Recognition based on Regulated Transformer and Spatio-Temporal Fusion Strategy for Driver Assistive Systems","code":"https://github.com/SMIL-SPCRAS/AVCRFormer","n_code_links":1,"syntology":null},{"rank_in_archive_order":5,"model":"3D Conv + EfficientNetV2 + Transformer + TCN","metrics":{"Top-1 Accuracy":"89.52"},"uses_additional_data":false,"paper_date":"2022-05-23","paper":"/paper/accurate-and-resource-efficient-lipreading","paper_url":"https://ieeexplore.ieee.org/document/9747729","paper_title":"Accurate and Resource-Efficient Lipreading with Efficientnetv2 and Transformers","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":6,"model":"Vosk + MediaPipe + LS + MixUp + SA + 3DResNet-18 + BiLSTM + Cosine WR","metrics":{"Top-1 Accuracy":"88.7"},"uses_additional_data":false,"paper_date":"2022-08-29","paper":"/paper/visual-speech-recognition-in-a-driver","paper_url":"https://eurasip.org/Proceedings/Eusipco/Eusipco2022/pdfs/0001131.pdf","paper_title":"Visual Speech Recognition in a Driver Assistance System","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":7,"model":"3D Conv + ResNet-18 + MS-TCN + Multi-Head Visual-Audio Memory","metrics":{"Top-1 Accuracy":"88.5"},"uses_additional_data":false,"paper_date":"2022-04-04","paper":"/paper/distinguishing-homophenes-using-multi-head-1","paper_url":"https://arxiv.org/abs/2204.01725v1","paper_title":"Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading","code":"https://github.com/ms-dot-k/Multi-head-Visual-Audio-Memory","n_code_links":1,"syntology":null},{"rank_in_archive_order":8,"model":"3D Conv + ResNet-18 + MS-TCN + KD (Ensemble)","metrics":{"Top-1 Accuracy":"88.5"},"uses_additional_data":false,"paper_date":"2020-07-13","paper":"/paper/towards-practical-lipreading-with-distilled","paper_url":"https://arxiv.org/abs/2007.06504v3","paper_title":"Towards Practical Lipreading with Distilled and Efficient Models","code":"https://github.com/mpc001/Lipreading_using_Temporal_Convolutional_Networks","n_code_links":1,"syntology":null},{"rank_in_archive_order":9,"model":"3D-ResNet + Bi-GRU + MixUp + Label Smoothing + Cosine LR (Word Boundary)","metrics":{"Top-1 Accuracy":"88.4"},"uses_additional_data":false,"paper_date":"2020-11-15","paper":"/paper/learn-an-effective-lip-reading-model-without","paper_url":"https://arxiv.org/abs/2011.07557v1","paper_title":"Learn an Effective Lip Reading Model without Pains","code":"https://github.com/Fengdalu/learn-an-effective-lip-reading-model-without-pains","n_code_links":1,"syntology":null},{"rank_in_archive_order":10,"model":"3D-ResNet + Bi-GRU + MixUp + Label Smoothing + Cosine LR","metrics":{"Top-1 Accuracy":"85.5"},"uses_additional_data":false,"paper_date":"2020-11-15","paper":"/paper/learn-an-effective-lip-reading-model-without","paper_url":"https://arxiv.org/abs/2011.07557v1","paper_title":"Learn an Effective Lip Reading Model without Pains","code":"https://github.com/Fengdalu/learn-an-effective-lip-reading-model-without-pains","n_code_links":1,"syntology":null},{"rank_in_archive_order":11,"model":"3D Conv + ResNet-18 + Bi-GRU + Visual-Audio Memory","metrics":{"Top-1 Accuracy":"85.4"},"uses_additional_data":false,"paper_date":"2022-04-04","paper":"/paper/multi-modality-associative-bridging-through-1","paper_url":"https://arxiv.org/abs/2204.01265v1","paper_title":"Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video","code":"https://github.com/ms-dot-k/Visual-Audio-Memory","n_code_links":1,"syntology":{"n_ran":1,"n_unverified":0,"n_samples":1,"n_pointer_only_licence":1}},{"rank_in_archive_order":12,"model":"3D Conv + ResNet-18 + MS-TCN","metrics":{"Top-1 Accuracy":"85.30"},"uses_additional_data":false,"paper_date":"2020-01-23","paper":"/paper/lipreading-using-temporal-convolutional","paper_url":"https://arxiv.org/abs/2001.08702v1","paper_title":"Lipreading using Temporal Convolutional Networks","code":"https://github.com/mpc001/Lipreading_using_Temporal_Convolutional_Networks","n_code_links":2,"syntology":{"n_ran":1,"n_unverified":1,"n_samples":2,"n_pointer_only_licence":2}},{"rank_in_archive_order":13,"model":"3D Conv + ResNet-18 + Bi-GRU(Face Cutout)","metrics":{"Top-1 Accuracy":"85.02"},"uses_additional_data":false,"paper_date":"2020-03-06","paper":"/paper/can-we-read-speech-beyond-the-lips-rethinking","paper_url":"https://arxiv.org/abs/2003.03206v2","paper_title":"Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition","code":"https://github.com/sailordiary/deep-face-vsr","n_code_links":1,"syntology":null},{"rank_in_archive_order":14,"model":"MoCo + Wav2Vec by SJTU LUMIA","metrics":{"Top-1 Accuracy":"85.0"},"uses_additional_data":false,"paper_date":"2022-02-24","paper":"/paper/leveraging-uni-modal-self-supervised-learning-1","paper_url":"https://arxiv.org/abs/2203.07996v2","paper_title":"Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition","code":"https://github.com/lumia-group/leveraging-self-supervised-learning-for-avsr","n_code_links":1,"syntology":{"n_ran":6,"n_unverified":0,"n_samples":6,"n_pointer_only_licence":0}},{"rank_in_archive_order":15,"model":"3D Conv + P3D-ResNet50 + TCN","metrics":{"Top-1 Accuracy":"84.80"},"uses_additional_data":false,"paper_date":"2020-05-12","paper":"/paper/discriminative-multi-modality-speech","paper_url":"https://arxiv.org/abs/2005.05592v2","paper_title":"Discriminative Multi-modality Speech Recognition","code":"https://github.com/JackSyu/Discriminative-Multi-modality-Speech-Recognition","n_code_links":2,"syntology":null},{"rank_in_archive_order":16,"model":"3D Conv + ResNet-18 + Bi-GRU","metrics":{"Top-1 Accuracy":"84.41"},"uses_additional_data":false,"paper_date":"2020-03-13","paper":"/paper/mutual-information-maximization-for-effective","paper_url":"https://arxiv.org/abs/2003.06439v1","paper_title":"Mutual Information Maximization for Effective Lip Reading","code":"https://github.com/xing96/MIM-lipreading","n_code_links":1,"syntology":null},{"rank_in_archive_order":17,"model":"SpotFast + Transformer + Product-Key memory","metrics":{"Top-1 Accuracy":"84.4"},"uses_additional_data":false,"paper_date":"2020-05-21","paper":"/paper/spotfast-networks-with-memory-augmented","paper_url":"https://arxiv.org/abs/2005.10903v1","paper_title":"SpotFast Networks with Memory Augmented Lateral Transformers for Lipreading","code":"https://github.com/perathambkk/lipreading","n_code_links":1,"syntology":null},{"rank_in_archive_order":18,"model":"DFTN","metrics":{"Top-1 Accuracy":"84.13"},"uses_additional_data":false,"paper_date":"2020-03-12","paper":"/paper/deformation-flow-based-two-stream-network-for","paper_url":"https://arxiv.org/abs/2003.05709v2","paper_title":"Deformation Flow Based Two-Stream Network for Lip Reading","code":"https://github.com/jingyunx/Deformation-Flow-Based-Two-stream-Network","n_code_links":1,"syntology":null},{"rank_in_archive_order":19,"model":"PCPG","metrics":{"Top-1 Accuracy":"83.5"},"uses_additional_data":false,"paper_date":"2020-03-09","paper":"/paper/pseudo-convolutional-policy-gradient-for","paper_url":"https://arxiv.org/abs/2003.03983v1","paper_title":"Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":20,"model":"3D Conv + ResNet-34 + Bi-GRU","metrics":{"Top-1 Accuracy":"83.39"},"uses_additional_data":false,"paper_date":"2018-02-18","paper":"/paper/end-to-end-audiovisual-speech-recognition","paper_url":"http://arxiv.org/abs/1802.06424v2","paper_title":"End-to-end Audiovisual Speech Recognition","code":"https://github.com/mpc001/end-to-end-Lipreading","n_code_links":2,"syntology":null},{"rank_in_archive_order":21,"model":"Multi-grained + Bi-ConvLSTM","metrics":{"Top-1 Accuracy":"83.34"},"uses_additional_data":false,"paper_date":"2019-08-30","paper":"/paper/multi-grained-spatio-temporal-modeling-for","paper_url":"https://arxiv.org/abs/1908.11618v2","paper_title":"Multi-Grained Spatio-temporal Modeling for Lip-reading","code":null,"n_code_links":0,"syntology":null},{"rank_in_archive_order":22,"model":"3D Conv + ResNet-34 + Bi-LSTM","metrics":{"Top-1 Accuracy":"83.00"},"uses_additional_data":false,"paper_date":"2017-03-12","paper":"/paper/combining-residual-networks-with-lstms-for","paper_url":"http://arxiv.org/abs/1703.04105v4","paper_title":"Combining Residual Networks with LSTMs for Lipreading","code":"https://github.com/tstafylakis/Lipreading-ResNet","n_code_links":4,"syntology":{"n_ran":2,"n_unverified":0,"n_samples":2,"n_pointer_only_licence":1}}],"since_archive":{"claim":"Results that newer papers report for their own method, placed here by Syntology. A model pointed at the cell in the paper's own table; the number was read from that cell and checked against this leaderboard's metric, dataset, split and scale; an independent check that saw this leaderboard's other rows and every other leaderboard on the same dataset accepted it. Not reviewed by the paper's authors or by the archive's editors, and not ranked against the archive rows.","extraction_file_present":true,"measurement":{"test_papers":883,"papers_with_output":881,"judged_true":108,"judged":110,"wilson95_lower":0.9361,"measured_on":"2026-09-24","frozen_commit":"0e3de0df94"},"measurement_note":"blind adjudication of accepted entries on a held-out split of archive papers, rules frozen before the test","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","complete":false,"papers_newer_than_archive":9623,"papers_checked":6885,"papers_extracted_not_yet_verified":0,"boards_without_verdict":2,"papers_not_yet_extracted":2737},"order":"newest first by month (arXiv date, else the arXiv-id month), then arXiv id descending","columns":[],"entries":[]},"syntology":{"read_at":"2026-09-25T09:33:49+00:00","claim":"Per row: N of M harvested code samples from that row's paper executed on a synthesized fixture; the other M-N are unverified. Not a reproduction of the row's number; not a correctness claim. n_pointer_only_licence counts samples the site points at rather than redistributes (a licence axis, independent of ran/unverified).","rows_with_graph_line":4,"rows_with_any_sample_ran":4,"distinct_papers_with_graph_line":4,"distinct_papers_with_any_sample_ran":4,"samples_over_distinct_papers":{"n_ran":10,"n_unverified":1,"n_samples":11,"n_pointer_only_licence":4,"note":"each paper (arXiv id) counted once, however many rows it is behind; this is the page-level figure"},"samples_row_weighted":{"n_ran":10,"n_unverified":1,"n_samples":11,"n_pointer_only_licence":4,"note":"row-weighted: a paper behind several rows is counted once per row; inflated relative to samples_over_distinct_papers by design, kept for readers summing the per-row syntology blocks"}}}