{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/epic-fusion-audio-visual-temporal-binding-for","title":"EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognition","arxiv_id":"1908.08498","date":"2019-08-22","proceeding":"ICCV 2019 10","authors":["Evangelos Kazakos","Arsha Nagrani","Andrew Zisserman","Dima Damen"],"abstract":"We focus on multi-modal fusion for egocentric action recognition, and propose a novel architecture for multi-modal temporal-binding, i.e. the combination of modalities within a range of temporal offsets. We train the architecture with three modalities -- RGB, Flow and Audio -- and combine them with mid-level fusion alongside sparse temporal sampling of fused representations. In contrast with previous works, modalities are fused before temporal aggregation, with shared modality and fusion weights over time. Our proposed architecture is trained end-to-end, outperforming individual modalities as well as late-fusion of modalities. We demonstrate the importance of audio in egocentric vision, on per-class basis, for identifying actions as well as interacting objects. Our method achieves state of the art results on both the seen and unseen test sets of the largest egocentric dataset: EPIC-Kitchens, on all metrics using the public leaderboard.","url_abs":"https://arxiv.org/abs/1908.08498v1","url_pdf":"https://arxiv.org/pdf/1908.08498v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"epic-fusion-audio-visual-temporal-binding-for","repo_url":"https://github.com/ekazakos/temporal-binding-network","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"egocentric-activity-recognition","task_name":"Egocentric Activity Recognition"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/egocentric-activity-recognition-on-epic-1","task":"Egocentric Activity Recognition","dataset":"EPIC-KITCHENS-55","model":"TBN","rank_in_archive_order":2,"of":7,"metrics":{"Actions Top-1 (S1)":"34.8","Actions Top-1 (S2)":"19.06"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1908.08498","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}