{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/end-to-end-temporal-action-detection-with","title":"End-to-end Temporal Action Detection with Transformer","arxiv_id":"2106.10271","date":"2021-06-18","proceeding":null,"authors":["Xiaolong Liu","Qimeng Wang","Yao Hu","Xu Tang","Shiwei Zhang","Song Bai","Xiang Bai"],"abstract":"Temporal action detection (TAD) aims to determine the semantic label and the temporal interval of every action instance in an untrimmed video. It is a fundamental and challenging task in video understanding. Previous methods tackle this task with complicated pipelines. They often need to train multiple networks and involve hand-designed operations, such as non-maximal suppression and anchor generation, which limit the flexibility and prevent end-to-end learning. In this paper, we propose an end-to-end Transformer-based method for TAD, termed TadTR. Given a small set of learnable embeddings called action queries, TadTR adaptively extracts temporal context information from the video for each query and directly predicts action instances with the context. To adapt Transformer to TAD, we propose three improvements to enhance its locality awareness. The core is a temporal deformable attention module that selectively attends to a sparse set of key snippets in a video. A segment refinement mechanism and an actionness regression head are designed to refine the boundaries and confidence of the predicted instances, respectively. With such a simple pipeline, TadTR requires lower computation cost than previous detectors, while preserving remarkable performance. As a self-contained detector, it achieves state-of-the-art performance on THUMOS14 (56.7% mAP) and HACS Segments (32.09% mAP). Combined with an extra action classifier, it obtains 36.75% mAP on ActivityNet-1.3. Code is available at https://github.com/xlliu7/TadTR.","url_abs":"https://arxiv.org/abs/2106.10271v4","url_pdf":"https://arxiv.org/pdf/2106.10271v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"end-to-end-temporal-action-detection-with","repo_url":"https://github.com/xlliu7/TadTR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"action-detection","task_name":"Action Detection"},{"task_slug":"action-recognition","task_name":"Temporal Action Localization"},{"task_slug":"video-understanding","task_name":"Video Understanding"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"deformable-attention-module","method_name":"Deformable Attention Module"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/temporal-action-localization-on-activitynet","task":"Temporal Action Localization","dataset":"ActivityNet-1.3","model":"TadTR (TSP features)","rank_in_archive_order":14,"of":33,"metrics":{"mAP":"36.75","mAP IOU@0.5":"53.62","mAP IOU@0.75":"37.52","mAP IOU@0.95":"10.56"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-hacs","task":"Temporal Action Localization","dataset":"HACS","model":"TadTr (I3D RGB)","rank_in_archive_order":10,"of":12,"metrics":{"Average-mAP":"32.09","mAP@0.5":"47.14","mAP@0.75":"32.11","mAP@0.95":"10.94"},"uses_additional_data":false},{"leaderboard":"/sota/temporal-action-localization-on-thumos14","task":"Temporal Action Localization","dataset":"THUMOS’14","model":"TadTR","rank_in_archive_order":16,"of":42,"metrics":{"Avg mAP (0.3:0.7)":"56.7","mAP IOU@0.3":"74.8","mAP IOU@0.4":"69.1","mAP IOU@0.5":"60.1","mAP IOU@0.6":"46.6","mAP IOU@0.7":"32.8"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2106.10271","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.10271"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xlliu7/TadTR","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f4bbeee84bc07a00","entry":"get_classes","repo":"xlliu7/TadTR","repo_kind":"official","path":"Evaluation/eval_detection.py","file_url":"https://github.com/xlliu7/TadTR/blob/HEAD/Evaluation/eval_detection.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f4bbeee84bc07a00"}},{"code_sha256_prefix":"483d79f60d8148d2","entry":"setup_logger","repo":"xlliu7/TadTR","repo_kind":"official","path":"Evaluation/eval_detection.py","file_url":"https://github.com/xlliu7/TadTR/blob/HEAD/Evaluation/eval_detection.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"483d79f60d8148d2"}},{"code_sha256_prefix":"2795b6049353a54d","entry":"compute_average_precision_detection","repo":"xlliu7/TadTR","repo_kind":"official","path":"Evaluation/eval_detection.py","file_url":"https://github.com/xlliu7/TadTR/blob/HEAD/Evaluation/eval_detection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2795b6049353a54d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}