{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lip-reading-sentences-in-the-wild","title":"Lip Reading Sentences in the Wild","arxiv_id":"1611.05358","date":"2016-11-16","proceeding":"CVPR 2017 7","authors":["Joon Son Chung","Andrew Senior","Oriol Vinyals","Andrew Zisserman"],"abstract":"The goal of this work is to recognise phrases and sentences being spoken by a\ntalking face, with or without the audio. Unlike previous works that have\nfocussed on recognising a limited number of words or phrases, we tackle lip\nreading as an open-world problem - unconstrained natural language sentences,\nand in the wild videos.\n  Our key contributions are: (1) a 'Watch, Listen, Attend and Spell' (WLAS)\nnetwork that learns to transcribe videos of mouth motion to characters; (2) a\ncurriculum learning strategy to accelerate training and to reduce overfitting;\n(3) a 'Lip Reading Sentences' (LRS) dataset for visual speech recognition,\nconsisting of over 100,000 natural sentences from British television.\n  The WLAS model trained on the LRS dataset surpasses the performance of all\nprevious work on standard lip reading benchmark datasets, often by a\nsignificant margin. This lip reading performance beats a professional lip\nreader on videos from BBC television, and we also demonstrate that visual\ninformation helps to improve speech recognition performance even when the audio\nis available.","url_abs":"http://arxiv.org/abs/1611.05358v2","url_pdf":"http://arxiv.org/pdf/1611.05358v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"lip-reading","task_name":"Lip Reading"},{"task_slug":"lipreading","task_name":"Lipreading"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"visual-speech-recognition","task_name":"Visual Speech Recognition"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[],"datasets_introduced":[{"slug":"lrs2","name":"LRS2","full_name":"Lip Reading Sentences 2"}],"methods_introduced":[],"results":[{"leaderboard":"/sota/lipreading-on-grid-corpus-mixed-speech","task":"Lipreading","dataset":"GRID corpus (mixed-speech)","model":"WAS","rank_in_archive_order":4,"of":5,"metrics":{"Word Error Rate (WER)":"3"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1611.05358","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}