{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/testa-temporal-spatial-token-aggregation-for","title":"TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding","arxiv_id":"2310.19060","date":"2023-10-29","proceeding":null,"authors":["Shuhuai Ren","Sishuo Chen","Shicheng Li","Xu sun","Lu Hou"],"abstract":"Large-scale video-language pre-training has made remarkable strides in advancing video-language understanding tasks. However, the heavy computational burden of video encoding remains a formidable efficiency bottleneck, particularly for long-form videos. These videos contain massive visual tokens due to their inherent 3D properties and spatiotemporal redundancy, making it challenging to capture complex temporal and spatial relationships. To tackle this issue, we propose an efficient method called TEmporal-Spatial Token Aggregation (TESTA). TESTA condenses video semantics by adaptively aggregating similar frames, as well as similar patches within each frame. TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding. Building upon TESTA, we introduce a pre-trained video-language model equipped with a divided space-time token aggregation module in each video encoder block. We evaluate our model on five datasets for paragraph-to-video retrieval and long-form VideoQA tasks. Experimental results show that TESTA improves computing efficiency by 1.7 times, and achieves significant performance gains from its scalability in processing longer input frames, e.g., +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie.","url_abs":"https://arxiv.org/abs/2310.19060v1","url_pdf":"https://arxiv.org/pdf/2310.19060v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"testa-temporal-spatial-token-aggregation-for","repo_url":"https://github.com/renshuhuai-andy/testa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"form","task_name":"Form"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"video-question-answering","task_name":"Video Question Answering"},{"task_slug":"video-retrieval","task_name":"Video Retrieval"},{"task_slug":"video-text-retrieval","task_name":"Video-Text Retrieval"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/video-question-answering-on-activitynet-qa","task":"Video Question Answering","dataset":"ActivityNet-QA","model":"TESTA (ViT-B/16)","rank_in_archive_order":17,"of":36,"metrics":{"Accuracy":"45"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-activitynet","task":"Video Retrieval","dataset":"ActivityNet","model":"TESTA (ViT-B/16)","rank_in_archive_order":12,"of":31,"metrics":{"text-to-video R@1":"54.8","text-to-video R@10":"89.6","text-to-video R@5":"80.8"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-condensed-movies","task":"Video Retrieval","dataset":"Condensed Movies","model":"TESTA (ViT-B/16)","rank_in_archive_order":1,"of":3,"metrics":{"text-to-video R@1":"24.9","text-to-video R@10":"55.1","text-to-video R@5":"46.5"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-didemo","task":"Video Retrieval","dataset":"DiDeMo","model":"TESTA (ViT-B/16)","rank_in_archive_order":8,"of":40,"metrics":{"text-to-video R@1":"61.2","text-to-video R@10":"91.5","text-to-video R@5":"87.2"},"uses_additional_data":true},{"leaderboard":"/sota/video-retrieval-on-queryd","task":"Video Retrieval","dataset":"QuerYD","model":"TESTA (ViT-B/16)","rank_in_archive_order":1,"of":5,"metrics":{"text-to-video R@1":"83.4","text-to-video R@10":"95.3","text-to-video R@5":"93.8"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.19060","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.19060"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/renshuhuai-andy/testa","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_honours":1,"unverified":4},"by_repo_kind":{"official":{"samples":15,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"87f39a589f593732","entry":"bipartite_soft_matching","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"testa/merge.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/testa/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"87f39a589f593732"}},{"code_sha256_prefix":"969e8250d805a154","entry":"do_nothing","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"testa/merge.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/testa/merge.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"969e8250d805a154"}},{"code_sha256_prefix":"3869ab58c2773dbd","entry":"generate_colormap","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"testa/vis.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/testa/vis.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3869ab58c2773dbd"}},{"code_sha256_prefix":"d3dbd19bd0c74374","entry":"get_custom_args","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"models/blip.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/models/blip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d3dbd19bd0c74374"}},{"code_sha256_prefix":"674be50d432e8846","entry":"kth_bipartite_soft_matching","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"testa/merge_original.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/testa/merge_original.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"674be50d432e8846"}},{"code_sha256_prefix":"307c022dd3d5678e","entry":"merge_wavg","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"testa/merge.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/testa/merge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"307c022dd3d5678e"}},{"code_sha256_prefix":"b42d3b0c5a43e6d2","entry":"original_bipartite_soft_matching","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"testa/merge_original.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/testa/merge_original.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b42d3b0c5a43e6d2"}},{"code_sha256_prefix":"6670faac3ea134bd","entry":"params_count","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"model_stats.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/model_stats.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6670faac3ea134bd"}},{"code_sha256_prefix":"e7a7d47996cb6fd6","entry":"parse_merging_type","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"testa/utils.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/testa/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7a7d47996cb6fd6"}},{"code_sha256_prefix":"94e8f254ce489a4b","entry":"parse_r","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"testa/utils.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/testa/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94e8f254ce489a4b"}},{"code_sha256_prefix":"aa5a91226269bf75","entry":"tile","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"models/testa_vqa.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/models/testa_vqa.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa5a91226269bf75"}},{"code_sha256_prefix":"0ec9fc2025c16f65","entry":"all_gather_with_grad","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"models/testa_retrieval.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/models/testa_retrieval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ec9fc2025c16f65"}},{"code_sha256_prefix":"eaef07a98678e866","entry":"benchmark","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"testa/utils.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/testa/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eaef07a98678e866"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"models/testa_retrieval.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/models/testa_retrieval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}},{"code_sha256_prefix":"6bb5f8f00704ac2f","entry":"make_visualization","repo":"renshuhuai-andy/testa","repo_kind":"official","path":"testa/vis.py","file_url":"https://github.com/renshuhuai-andy/testa/blob/HEAD/testa/vis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bb5f8f00704ac2f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}