{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/revisiting-the-integration-of-convolution-and","title":"Revisiting the Integration of Convolution and Attention for Vision Backbone","arxiv_id":"2411.14429","date":"2024-11-21","proceeding":null,"authors":["Lei Zhu","Xinjiang Wang","Wayne Zhang","Rynson W. H. Lau"],"abstract":"Convolutions (Convs) and multi-head self-attentions (MHSAs) are typically considered alternatives to each other for building vision backbones. Although some works try to integrate both, they apply the two operators simultaneously at the finest pixel granularity. With Convs responsible for per-pixel feature extraction already, the question is whether we still need to include the heavy MHSAs at such a fine-grained level. In fact, this is the root cause of the scalability issue w.r.t. the input resolution for vision transformers. To address this important problem, we propose in this work to use MSHAs and Convs in parallel \\textbf{at different granularity levels} instead. Specifically, in each layer, we use two different ways to represent an image: a fine-grained regular grid and a coarse-grained set of semantic slots. We apply different operations to these two representations: Convs to the grid for local features, and MHSAs to the slots for global features. A pair of fully differentiable soft clustering and dispatching modules is introduced to bridge the grid and set representations, thus enabling local-global fusion. Through extensive experiments on various vision tasks, we empirically verify the potential of the proposed integration scheme, named \\textit{GLMix}: by offloading the burden of fine-grained features to light-weight Convs, it is sufficient to use MHSAs in a few (e.g., 64) semantic slots to match the performance of recent state-of-the-art backbones, while being more efficient. Our visualization results also demonstrate that the soft clustering module produces a meaningful semantic grouping effect with only IN1k classification supervision, which may induce better interpretability and inspire new weakly-supervised semantic segmentation approaches. Code will be available at \\url{https://github.com/rayleizhu/GLMix}.","url_abs":"https://arxiv.org/abs/2411.14429v1","url_pdf":"https://arxiv.org/pdf/2411.14429v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"revisiting-the-integration-of-convolution-and","repo_url":"https://github.com/rayleizhu/glmix","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"weakly-supervised-semantic-segmentation-1","task_name":"Weakly supervised Semantic Segmentation"},{"task_slug":"weakly-supervised-semantic-segmentation","task_name":"Weakly-Supervised Semantic Segmentation"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.14429","atlas_url":"https://app.syntology.ai/?focus=2411.14429","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.14429"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/rayleizhu/GLMix","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/rayleizhu/glmix","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_honours":3,"ran":3,"unverified":10},"by_repo_kind":{"official":{"samples":16,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"410bffb8a07b52e3","entry":"count_params","repo":"rayleizhu/GLMix","repo_kind":"official","path":"benchmark.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/benchmark.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":2,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"410bffb8a07b52e3"}},{"code_sha256_prefix":"750a1fb4a8be9fd9","entry":"GLMixBlock","repo":"rayleizhu/glmix","repo_kind":"official","path":"models/glnet.py","file_url":"https://github.com/rayleizhu/glmix/blob/HEAD/models/glnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"750a1fb4a8be9fd9"}},{"code_sha256_prefix":"3f3123c5b41c1bda","entry":"LayerScale","repo":"rayleizhu/glmix","repo_kind":"official","path":"models/glnet.py","file_url":"https://github.com/rayleizhu/glmix/blob/HEAD/models/glnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f3123c5b41c1bda"}},{"code_sha256_prefix":"7eca7076f186820a","entry":"ResDWConvNCHW","repo":"rayleizhu/glmix","repo_kind":"official","path":"models/glnet.py","file_url":"https://github.com/rayleizhu/glmix/blob/HEAD/models/glnet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7eca7076f186820a"}},{"code_sha256_prefix":"50d653ee6a59955c","entry":"cuda_timestamp","repo":"rayleizhu/GLMix","repo_kind":"official","path":"benchmark.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/benchmark.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"50d653ee6a59955c"}},{"code_sha256_prefix":"475160c5a1a1225a","entry":"timestamp","repo":"rayleizhu/GLMix","repo_kind":"official","path":"benchmark.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/benchmark.py","link_basis":"plan_row","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"475160c5a1a1225a"}},{"code_sha256_prefix":"9153241c4331eb1a","entry":"build_dataset","repo":"rayleizhu/GLMix","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/datasets.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9153241c4331eb1a"}},{"code_sha256_prefix":"3bc74137ab36aa79","entry":"build_transform","repo":"rayleizhu/GLMix","repo_kind":"official","path":"datasets.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/datasets.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3bc74137ab36aa79"}},{"code_sha256_prefix":"f4e3f81ce11955a1","entry":"dumps_pyarrow","repo":"rayleizhu/GLMix","repo_kind":"official","path":"folder2lmdb.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/folder2lmdb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4e3f81ce11955a1"}},{"code_sha256_prefix":"6ab1c8b0afb86428","entry":"evaluate","repo":"rayleizhu/GLMix","repo_kind":"official","path":"ConvNextV2/engine_finetune.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/ConvNextV2/engine_finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6ab1c8b0afb86428"}},{"code_sha256_prefix":"41f6df8410217556","entry":"get_dist_url","repo":"rayleizhu/GLMix","repo_kind":"official","path":"slurm_wrapper.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/slurm_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"41f6df8410217556"}},{"code_sha256_prefix":"4229711f1a52b2cd","entry":"get_init_file","repo":"rayleizhu/GLMix","repo_kind":"official","path":"slurm_wrapper.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/slurm_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4229711f1a52b2cd"}},{"code_sha256_prefix":"d07269a1ef6fc528","entry":"get_shared_folder","repo":"rayleizhu/GLMix","repo_kind":"official","path":"slurm_wrapper.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/slurm_wrapper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d07269a1ef6fc528"}},{"code_sha256_prefix":"6b3e468a3a7b483b","entry":"prefetch","repo":"rayleizhu/GLMix","repo_kind":"official","path":"prefetch_generator.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/prefetch_generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b3e468a3a7b483b"}},{"code_sha256_prefix":"bd61d1e4d25b3c45","entry":"rand_bbox","repo":"rayleizhu/GLMix","repo_kind":"official","path":"models/glnet_tklb.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/models/glnet_tklb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd61d1e4d25b3c45"}},{"code_sha256_prefix":"51738010a21f76a1","entry":"raw_reader","repo":"rayleizhu/GLMix","repo_kind":"official","path":"folder2lmdb.py","file_url":"https://github.com/rayleizhu/GLMix/blob/HEAD/folder2lmdb.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"51738010a21f76a1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}