{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/attention-based-audio-visual-fusion-for","title":"Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition","arxiv_id":"1809.01728","date":"2018-09-05","proceeding":null,"authors":["George Sterpu","Christian Saam","Naomi Harte"],"abstract":"Automatic speech recognition can potentially benefit from the lip motion\npatterns, complementing acoustic speech to improve the overall recognition\nperformance, particularly in noise. In this paper we propose an audio-visual\nfusion strategy that goes beyond simple feature concatenation and learns to\nautomatically align the two modalities, leading to enhanced representations\nwhich increase the recognition accuracy in both clean and noisy conditions. We\ntest our strategy on the TCD-TIMIT and LRS2 datasets, designed for large\nvocabulary continuous speech recognition, applying three types of noise at\ndifferent power ratios. We also exploit state of the art Sequence-to-Sequence\narchitectures, showing that our method can be easily integrated. Results show\nrelative improvements from 7% up to 30% on TCD-TIMIT over the acoustic modality\nalone, depending on the acoustic noise level. We anticipate that the fusion\nstrategy can easily generalise to many other multimodal tasks which involve\ncorrelated modalities. Code available online on GitHub:\nhttps://github.com/georgesterpu/Sigmedia-AVSR","url_abs":"http://arxiv.org/abs/1809.01728v3","url_pdf":"http://arxiv.org/pdf/1809.01728v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"attention-based-audio-visual-fusion-for","repo_url":"https://github.com/georgesterpu/Sigmedia-AVSR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"attention-based-audio-visual-fusion-for","repo_url":"https://github.com/georgesterpu/Taris","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"attention-based-audio-visual-fusion-for","repo_url":"https://github.com/georgesterpu/avsr-tf1","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null}],"tasks":[{"task_slug":"automatic-speech-recognition-2","task_name":"Automatic Speech Recognition"},{"task_slug":"automatic-speech-recognition","task_name":"Automatic Speech Recognition (ASR)"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1809.01728","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}