{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/tr-detr-task-reciprocal-transformer-for-joint","title":"TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detection","arxiv_id":"2401.02309","date":"2024-01-04","proceeding":null,"authors":["Hao Sun","Mingyao Zhou","Wenjing Chen","Wei Xie"],"abstract":"Video moment retrieval (MR) and highlight detection (HD) based on natural language queries are two highly related tasks, which aim to obtain relevant moments within videos and highlight scores of each video clip. Recently, several methods have been devoted to building DETR-based networks to solve both MR and HD jointly. These methods simply add two separate task heads after multi-modal feature extraction and feature interaction, achieving good performance. Nevertheless, these approaches underutilize the reciprocal relationship between two tasks. In this paper, we propose a task-reciprocal transformer based on DETR (TR-DETR) that focuses on exploring the inherent reciprocity between MR and HD. Specifically, a local-global multi-modal alignment module is first built to align features from diverse modalities into a shared latent space. Subsequently, a visual feature refinement is designed to eliminate query-irrelevant information from visual features for modal interaction. Finally, a task cooperation module is constructed to refine the retrieval pipeline and the highlight score prediction process by utilizing the reciprocity between MR and HD. Comprehensive experiments on QVHighlights, Charades-STA and TVSum datasets demonstrate that TR-DETR outperforms existing state-of-the-art methods. Codes are available at \\url{https://github.com/mingyao1120/TR-DETR}.","url_abs":"https://arxiv.org/abs/2401.02309v2","url_pdf":"https://arxiv.org/pdf/2401.02309v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"tr-detr-task-reciprocal-transformer-for-joint","repo_url":"https://github.com/mingyao1120/tr-detr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"highlight-detection","task_name":"Highlight Detection"},{"task_slug":"moment-retrieval","task_name":"Moment Retrieval"},{"task_slug":"natural-language-queries","task_name":"Natural Language Queries"},{"task_slug":"retrieval","task_name":"Retrieval"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"detr","method_name":"Detr"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"feedforward-network","method_name":"Feedforward Network"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2401.02309","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2401.02309"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mingyao1120/tr-detr","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/mingyao1120/TR-DETR","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":8,"ran_draft_wrong":1,"ran_honours":1,"ran_violates":1,"unverified":6},"by_repo_kind":{"official":{"samples":17,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":17,"samples":[{"code_sha256_prefix":"07c665a133219e9a","entry":"CQAttention","repo":"mingyao1120/TR-DETR","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/TR-DETR/blob/HEAD/tr_detr/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"07c665a133219e9a"}},{"code_sha256_prefix":"e5976903cd2d055b","entry":"CQConcatenate","repo":"mingyao1120/TR-DETR","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/TR-DETR/blob/HEAD/tr_detr/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e5976903cd2d055b"}},{"code_sha256_prefix":"9ede126dc91c3fb3","entry":"Conv1D","repo":"mingyao1120/TR-DETR","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/TR-DETR/blob/HEAD/tr_detr/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9ede126dc91c3fb3"}},{"code_sha256_prefix":"2b35f8edda8e24ff","entry":"GRUFeatureExtractor","repo":"mingyao1120/TR-DETR","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/TR-DETR/blob/HEAD/tr_detr/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"2b35f8edda8e24ff"}},{"code_sha256_prefix":"4abf288345162e2d","entry":"LinearLayer","repo":"mingyao1120/TR-DETR","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/TR-DETR/blob/HEAD/tr_detr/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"4abf288345162e2d"}},{"code_sha256_prefix":"0b3ec8602f06002c","entry":"WeightedPool","repo":"mingyao1120/TR-DETR","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/TR-DETR/blob/HEAD/tr_detr/model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0b3ec8602f06002c"}},{"code_sha256_prefix":"d6ca656c20df137a","entry":"build_position_encoding","repo":"mingyao1120/tr-detr","repo_kind":"official","path":"tr_detr/position_encoding.py","file_url":"https://github.com/mingyao1120/tr-detr/blob/HEAD/tr_detr/position_encoding.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d6ca656c20df137a"}},{"code_sha256_prefix":"6897880b6ce14c94","entry":"inverse_sigmoid","repo":"mingyao1120/tr-detr","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/tr-detr/blob/HEAD/tr_detr/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6897880b6ce14c94"}},{"code_sha256_prefix":"60df4d5a1eee1b92","entry":"mask_logits","repo":"mingyao1120/TR-DETR","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/TR-DETR/blob/HEAD/tr_detr/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"60df4d5a1eee1b92"}},{"code_sha256_prefix":"ca2b2a4eafc4128b","entry":"prepare_batch_inputs","repo":"mingyao1120/tr-detr","repo_kind":"official","path":"tr_detr/start_end_dataset.py","file_url":"https://github.com/mingyao1120/tr-detr/blob/HEAD/tr_detr/start_end_dataset.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ca2b2a4eafc4128b"}},{"code_sha256_prefix":"b7639c2e8b83837b","entry":"span_cxw_to_xx","repo":"mingyao1120/TR-DETR","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/TR-DETR/blob/HEAD/tr_detr/model.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"b7639c2e8b83837b"}},{"code_sha256_prefix":"aae412c1aa042138","entry":"TRDETR","repo":"mingyao1120/TR-DETR","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/TR-DETR/blob/HEAD/tr_detr/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"aae412c1aa042138"}},{"code_sha256_prefix":"30ea9d77ca415e66","entry":"VSLFuser","repo":"mingyao1120/TR-DETR","repo_kind":"official","path":"tr_detr/model.py","file_url":"https://github.com/mingyao1120/TR-DETR/blob/HEAD/tr_detr/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"30ea9d77ca415e66"}},{"code_sha256_prefix":"081c8abd57d082fd","entry":"accuracy","repo":"mingyao1120/tr-detr","repo_kind":"official","path":"tr_detr/misc.py","file_url":"https://github.com/mingyao1120/tr-detr/blob/HEAD/tr_detr/misc.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"081c8abd57d082fd"}},{"code_sha256_prefix":"a82091666b9db8cf","entry":"multi_head_attention_forward","repo":"mingyao1120/tr-detr","repo_kind":"official","path":"tr_detr/attention.py","file_url":"https://github.com/mingyao1120/tr-detr/blob/HEAD/tr_detr/attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a82091666b9db8cf"}},{"code_sha256_prefix":"7114f025044af9b4","entry":"span_xx_to_cxw","repo":"mingyao1120/tr-detr","repo_kind":"official","path":"tr_detr/span_utils.py","file_url":"https://github.com/mingyao1120/tr-detr/blob/HEAD/tr_detr/span_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7114f025044af9b4"}},{"code_sha256_prefix":"e4ae5582a2533cdb","entry":"temporal_iou","repo":"mingyao1120/tr-detr","repo_kind":"official","path":"tr_detr/span_utils.py","file_url":"https://github.com/mingyao1120/tr-detr/blob/HEAD/tr_detr/span_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4ae5582a2533cdb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}