{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/holistic-interaction-transformer-network-for","title":"Holistic Interaction Transformer Network for Action Detection","arxiv_id":"2210.12686","date":"2022-10-23","proceeding":null,"authors":["Gueter Josmy Faure","Min-Hung Chen","Shang-Hong Lai"],"abstract":"Actions are about how we interact with the environment, including other people, objects, and ourselves. In this paper, we propose a novel multi-modal Holistic Interaction Transformer Network (HIT) that leverages the largely ignored, but critical hand and pose information essential to most human actions. The proposed \"HIT\" network is a comprehensive bi-modal framework that comprises an RGB stream and a pose stream. Each of them separately models person, object, and hand interactions. Within each sub-network, an Intra-Modality Aggregation module (IMA) is introduced that selectively merges individual interaction units. The resulting features from each modality are then glued using an Attentive Fusion Mechanism (AFM). Finally, we extract cues from the temporal context to better classify the occurring actions using cached memory. Our method significantly outperforms previous approaches on the J-HMDB, UCF101-24, and MultiSports datasets. We also achieve competitive results on AVA. The code will be available at https://github.com/joslefaure/HIT.","url_abs":"https://arxiv.org/abs/2210.12686v2","url_pdf":"https://arxiv.org/pdf/2210.12686v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"holistic-interaction-transformer-network-for","repo_url":"https://github.com/joslefaure/hit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"action-detection","task_name":"Action Detection"},{"task_slug":"action-recognition-in-videos","task_name":"Action Recognition"},{"task_slug":"fine-grained-action-detection","task_name":"Fine-Grained Action Detection"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/action-detection-on-j-hmdb","task":"Action Detection","dataset":"J-HMDB","model":"HIT","rank_in_archive_order":2,"of":18,"metrics":{"Frame-mAP 0.5":"83.8","Video-mAP 0.2":"89.7","Video-mAP 0.5":"88.1"},"uses_additional_data":false},{"leaderboard":"/sota/action-detection-on-multisports","task":"Action Detection","dataset":"MultiSports","model":"HIT","rank_in_archive_order":1,"of":2,"metrics":{"Frame-mAP 0.5":"33.3","Video-mAP 0.2":"27.8","Video-mAP 0.5":"8.8"},"uses_additional_data":false},{"leaderboard":"/sota/action-detection-on-ucf101-24","task":"Action Detection","dataset":"UCF101-24","model":"HIT","rank_in_archive_order":4,"of":19,"metrics":{"Frame-mAP 0.5":"84.8","Video-mAP 0.2":"88.8","Video-mAP 0.5":"74.3"},"uses_additional_data":false},{"leaderboard":"/sota/action-recognition-on-ava-v2-2","task":"Action Recognition","dataset":"AVA v2.2","model":"HIT","rank_in_archive_order":22,"of":38,"metrics":{"mAP":"32.6"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.12686","mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}