Papers › Holistic Interaction Transformer Network for Action Detection

Holistic Interaction Transformer Network for Action Detection

23 Oct 2022arXiv:2210.12686archive 2025-07-28

Gueter Josmy Faure, Min-Hung Chen, Shang-Hong Lai

Actions are about how we interact with the environment, including other people, objects, and ourselves. In this paper, we propose a novel multi-modal Holistic Interaction Transformer Network (HIT) that leverages the largely ignored, but critical hand and pose information essential to most human actions. The proposed "HIT" network is a comprehensive bi-modal framework that comprises an RGB stream and a pose stream. Each of them separately models person, object, and hand interactions. Within each sub-network, an Intra-Modality Aggregation module (IMA) is introduced that selectively merges individual interaction units. The resulting features from each modality are then glued using an Attentive Fusion Mechanism (AFM). Finally, we extract cues from the temporal context to better classify the occurring actions using cached memory. Our method significantly outperforms previous approaches on the J-HMDB, UCF101-24, and MultiSports datasets. We also achieve competitive results on AVA. The code will be available at https://github.com/joslefaure/HIT.

Paper PDF Code

In Syntology Open this paper in Syntology's Atlas, the map of the papers in Syntology's graph and their citations.

Code

joslefaure/hit officialmentioned in papermentioned on GitHub report

Repository list and official/mentioned flags are the archive's, frozen 2025-07-28. Reachability, where shown, is from one Syntology probe window (2026-09-16 to 2026-09-18); repositories not probed show nothing. GitHub stars are not tracked.

Code Syntology ran Syntology

Not run by Syntology. Nothing on this page verifies that the listed code works.

Tasks

Action Detection Action Recognition Fine-Grained Action Detection Temporal Action Localization

Results from the paper archive 2025-07-28

Task	Dataset	Model	Metric	Value	Rank at snapshot	Leaderboard	Report
Action Detection	J-HMDB	HIT	Frame-mAP 0.5	83.8	#2 of 18	Archive leaderboard	report
Action Detection	J-HMDB	HIT	Video-mAP 0.2	89.7	#2 of 18	Archive leaderboard	report
Action Detection	J-HMDB	HIT	Video-mAP 0.5	88.1	#2 of 18	Archive leaderboard	report
Action Detection	MultiSports	HIT	Frame-mAP 0.5	33.3	#1 of 2	Archive leaderboard	report
Action Detection	MultiSports	HIT	Video-mAP 0.2	27.8	#1 of 2	Archive leaderboard	report
Action Detection	MultiSports	HIT	Video-mAP 0.5	8.8	#1 of 2	Archive leaderboard	report
Action Detection	UCF101-24	HIT	Frame-mAP 0.5	84.8	#4 of 19	Archive leaderboard	report
Action Detection	UCF101-24	HIT	Video-mAP 0.2	88.8	#4 of 19	Archive leaderboard	report
Action Detection	UCF101-24	HIT	Video-mAP 0.5	74.3	#4 of 19	Archive leaderboard	report
Action Recognition	AVA v2.2	HIT	mAP	32.6	#22 of 38	Archive leaderboard	report

Ranks are positions in the archive's leaderboards as they stood at the 2025-07-28 snapshot. Results published since then are not among these rows, so a rank here is not a current standing.

Methods

Absolute Position Encodings Adam Attention BPE Dense Connections Dropout Label Smoothing Layer Normalization Linear Layer Multi-Head Attention Position-Wise Feed-Forward Layer Residual Connection Softmax Transformer

Report a problem or propose a change · a person checks every report against the paper or source before anything changes; decisions are listed on /corrections