{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/interaction-visual-transformer-for-egocentric","title":"Interaction Region Visual Transformer for Egocentric Action Anticipation","arxiv_id":"2211.14154","date":"2022-11-25","proceeding":"IEEE WACV 2024 1","authors":["Debaditya Roy","Ramanathan Rajendiran","Basura Fernando"],"abstract":"Human-object interaction is one of the most important visual cues and we propose a novel way to represent human-object interactions for egocentric action anticipation. We propose a novel transformer variant to model interactions by computing the change in the appearance of objects and human hands due to the execution of the actions and use those changes to refine the video representation. Specifically, we model interactions between hands and objects using Spatial Cross-Attention (SCA) and further infuse contextual information using Trajectory Cross-Attention to obtain environment-refined interaction tokens. Using these tokens, we construct an interaction-centric video representation for action anticipation. We term our model InAViT which achieves state-of-the-art action anticipation performance on large-scale egocentric datasets EPICKTICHENS100 (EK100) and EGTEA Gaze+. InAViT outperforms other visual transformer-based methods including object-centric video representation. On the EK100 evaluation server, InAViT is the top-performing method on the public leaderboard (at the time of submission) where it outperforms the second-best model by 3.3% on mean-top5 recall.","url_abs":"https://arxiv.org/abs/2211.14154v7","url_pdf":"https://arxiv.org/pdf/2211.14154v7.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"interaction-visual-transformer-for-egocentric","repo_url":"https://github.com/lahaproject/inavit","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-anticipation","task_name":"Action Anticipation"},{"task_slug":"human-object-interaction-detection","task_name":"Human-Object Interaction Detection"},{"task_slug":"object","task_name":"Object"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-anticipation-on-egtea","task":"Action Anticipation","dataset":"EGTEA","model":"InAViT","rank_in_archive_order":2,"of":3,"metrics":{"Top-1 Accuracy":"67.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-anticipation-on-epic-kitchens-100","task":"Action Anticipation","dataset":"EPIC-KITCHENS-100","model":"InAViT","rank_in_archive_order":2,"of":9,"metrics":{"Recall@5":"25.89"},"uses_additional_data":false},{"leaderboard":"/sota/action-anticipation-on-epic-kitchens-100-test","task":"Action Anticipation","dataset":"EPIC-KITCHENS-100 (test)","model":"InAViT","rank_in_archive_order":1,"of":8,"metrics":{"recall@5":"23.75"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2211.14154","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}