{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adaptive-semantic-spatio-temporal-graph","title":"Adaptive Semantic-Spatio-Temporal Graph Convolutional Network for Lip Reading","arxiv_id":null,"date":"2021-08-16","proceeding":"IEEE Transactions on Multimedia 2021 8","authors":["Changchong Sheng","Xinzhong Zhu","Huiying Xu","Matti Pietikäinen","Li Liu"],"abstract":"The goal of this work is to recognize words, phrases, and sentences being spoken by a talking face without given the audio. Current deep learning approaches for lip reading focus on exploring the appearance and optical flow information of videos. However, these methods do not fully exploit the characteristics of lip motion. In addition to appearance and optical flow, the mouth contour deformation usually conveys significant information that is complementary to others. However, the modeling of dynamic mouth contour has received little attention than that of appearance and optical flow. In this work, we propose a novel model of dynamic mouth contours called Adaptive Semantic-Spatio-Temporal Graph Convolution Network (ASST-GCN), to go beyond previous methods by automatically learning both the spatial and temporal information from videos. To combine the complementary information from appearance and mouth contour, a two-stream visual front-end network is proposed. Experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art lip reading methods on several large-scale lip reading benchmarks.","url_abs":"https://ieeexplore.ieee.org/abstract/document/9514437?casa_token=PvNGnFvTIKMAAAAA:fp-CQSH9oHbVuasowYGyAAV8kVHaFJwFuePJPugpRqzFfrSxyuL6SIxYyQHzrJKopThmSFd1OA","url_pdf":"https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9514437","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"landmark-based-lipreading","task_name":"Landmark-based Lipreading"},{"task_slug":"lip-reading","task_name":"Lip Reading"},{"task_slug":"optical-flow-estimation","task_name":"Optical Flow Estimation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/landmark-based-lipreading-on-lrw","task":"Landmark-based Lipreading","dataset":"LRW","model":"Adaptive GCN","rank_in_archive_order":4,"of":5,"metrics":{"Top 1 Accuracy":"60.7"},"uses_additional_data":false}],"syntology":{"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}