{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/disentangling-spatial-and-temporal-learning","title":"Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning","arxiv_id":"2309.07911","date":"2023-09-14","proceeding":"ICCV 2023 1","authors":["Zhiwu Qing","Shiwei Zhang","Ziyuan Huang","Yingya Zhang","Changxin Gao","Deli Zhao","Nong Sang"],"abstract":"Recently, large-scale pre-trained language-image models like CLIP have shown extraordinary capabilities for understanding spatial contents, but naively transferring such models to video recognition still suffers from unsatisfactory temporal modeling capabilities. Existing methods insert tunable structures into or in parallel with the pre-trained model, which either requires back-propagation through the whole pre-trained model and is thus resource-demanding, or is limited by the temporal reasoning capability of the pre-trained structure. In this work, we present DiST, which disentangles the learning of spatial and temporal aspects of videos. Specifically, DiST uses a dual-encoder structure, where a pre-trained foundation model acts as the spatial encoder, and a lightweight network is introduced as the temporal encoder. An integration branch is inserted between the encoders to fuse spatio-temporal information. The disentangled spatial and temporal learning in DiST is highly efficient because it avoids the back-propagation of massive pre-trained parameters. Meanwhile, we empirically show that disentangled learning with an extra network for integration benefits both spatial and temporal understanding. Extensive experiments on five benchmarks show that DiST delivers better performance than existing state-of-the-art methods by convincing gaps. When pre-training on the large-scale Kinetics-710, we achieve 89.7% on Kinetics-400 with a frozen ViT-L model, which verifies the scalability of DiST. Codes and models can be found in https://github.com/alibaba-mmai-research/DiST.","url_abs":"https://arxiv.org/abs/2309.07911v1","url_pdf":"https://arxiv.org/pdf/2309.07911v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"disentangling-spatial-and-temporal-learning","repo_url":"https://github.com/alibaba-mmai-research/dist","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"video-recognition","task_name":"Video Recognition"}],"methods":[{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2309.07911","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2309.07911"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alibaba-mmai-research/dist","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/alibaba-mmai-research/DiST","reach":{"status":"ok"}}],"summary":{"ran":9,"unverified":1},"by_repo_kind":{"official":{"samples":10,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"069414abaa0b85d7","entry":"CrossAttentionBlockGenral","repo":"alibaba-mmai-research/DiST","repo_kind":"official","path":"models/module_zoo/branches/dist.py","file_url":"https://github.com/alibaba-mmai-research/DiST/blob/HEAD/models/module_zoo/branches/dist.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"069414abaa0b85d7"}},{"code_sha256_prefix":"107702c5e10f0772","entry":"Integration2TemporalNetwork","repo":"alibaba-mmai-research/DiST","repo_kind":"official","path":"models/module_zoo/branches/dist.py","file_url":"https://github.com/alibaba-mmai-research/DiST/blob/HEAD/models/module_zoo/branches/dist.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"107702c5e10f0772"}},{"code_sha256_prefix":"870afdf9c6d186fc","entry":"IntegrationNetwork","repo":"alibaba-mmai-research/DiST","repo_kind":"official","path":"models/module_zoo/branches/dist.py","file_url":"https://github.com/alibaba-mmai-research/DiST/blob/HEAD/models/module_zoo/branches/dist.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"870afdf9c6d186fc"}},{"code_sha256_prefix":"9ef28adc36df32e1","entry":"SpatialTemporalAdaPoolingNetwork","repo":"alibaba-mmai-research/DiST","repo_kind":"official","path":"models/module_zoo/branches/dist.py","file_url":"https://github.com/alibaba-mmai-research/DiST/blob/HEAD/models/module_zoo/branches/dist.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ef28adc36df32e1"}},{"code_sha256_prefix":"962982da044f81c4","entry":"Temporal2IntegrationNetwork","repo":"alibaba-mmai-research/DiST","repo_kind":"official","path":"models/module_zoo/branches/dist.py","file_url":"https://github.com/alibaba-mmai-research/DiST/blob/HEAD/models/module_zoo/branches/dist.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"962982da044f81c4"}},{"code_sha256_prefix":"010f261397fd25ae","entry":"TemporalNet","repo":"alibaba-mmai-research/DiST","repo_kind":"official","path":"models/module_zoo/branches/dist.py","file_url":"https://github.com/alibaba-mmai-research/DiST/blob/HEAD/models/module_zoo/branches/dist.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"010f261397fd25ae"}},{"code_sha256_prefix":"f450abfe243be2a1","entry":"drop_path","repo":"alibaba-mmai-research/dist","repo_kind":"official","path":"models/base/base_blocks.py","file_url":"https://github.com/alibaba-mmai-research/dist/blob/HEAD/models/base/base_blocks.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f450abfe243be2a1"}},{"code_sha256_prefix":"790ff3bdfa86f045","entry":"get_sinusoid_encoding_table","repo":"alibaba-mmai-research/dist","repo_kind":"official","path":"models/base/vit_video.py","file_url":"https://github.com/alibaba-mmai-research/dist/blob/HEAD/models/base/vit_video.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"790ff3bdfa86f045"}},{"code_sha256_prefix":"d78712f03ac90099","entry":"rename_model_state","repo":"alibaba-mmai-research/dist","repo_kind":"official","path":"process_dist_cpkt.py","file_url":"https://github.com/alibaba-mmai-research/dist/blob/HEAD/process_dist_cpkt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d78712f03ac90099"}},{"code_sha256_prefix":"f35fb0f6e3082e9e","entry":"DiSTNetwork","repo":"alibaba-mmai-research/DiST","repo_kind":"official","path":"models/module_zoo/branches/dist.py","file_url":"https://github.com/alibaba-mmai-research/DiST/blob/HEAD/models/module_zoo/branches/dist.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f35fb0f6e3082e9e"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}