{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/cost-aggregation-with-4d-convolutional-swin","title":"Cost Aggregation with 4D Convolutional Swin Transformer for Few-Shot Segmentation","arxiv_id":"2207.10866","date":"2022-07-22","proceeding":null,"authors":["Sunghwan Hong","Seokju Cho","Jisu Nam","Stephen Lin","Seungryong Kim"],"abstract":"This paper presents a novel cost aggregation network, called Volumetric Aggregation with Transformers (VAT), for few-shot segmentation. The use of transformers can benefit correlation map aggregation through self-attention over a global receptive field. However, the tokenization of a correlation map for transformer processing can be detrimental, because the discontinuity at token boundaries reduces the local context available near the token edges and decreases inductive bias. To address this problem, we propose a 4D Convolutional Swin Transformer, where a high-dimensional Swin Transformer is preceded by a series of small-kernel convolutions that impart local context to all pixels and introduce convolutional inductive bias. We additionally boost aggregation performance by applying transformers within a pyramidal structure, where aggregation at a coarser level guides aggregation at a finer level. Noise in the transformer output is then filtered in the subsequent decoder with the help of the query's appearance embedding. With this model, a new state-of-the-art is set for all the standard benchmarks in few-shot segmentation. It is shown that VAT attains state-of-the-art performance for semantic correspondence as well, where cost aggregation also plays a central role.","url_abs":"https://arxiv.org/abs/2207.10866v1","url_pdf":"https://arxiv.org/pdf/2207.10866v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"cost-aggregation-with-4d-convolutional-swin","repo_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"few-shot-image-segmentation","task_name":"Few-Shot Semantic Segmentation"},{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"semantic-correspondence","task_name":"Semantic correspondence"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"stochastic-depth","method_name":"Stochastic Depth"},{"method_slug":"swin-transformer","method_name":"Swin Transformer"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/few-shot-semantic-segmentation-on-coco-20i-1","task":"Few-Shot Semantic Segmentation","dataset":"COCO-20i (1-shot)","model":"VAT (ResNet-101)","rank_in_archive_order":58,"of":85,"metrics":{"FB-IoU":"68.8","Mean IoU":"41.3"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-semantic-segmentation-on-coco-20i-5","task":"Few-Shot Semantic Segmentation","dataset":"COCO-20i (5-shot)","model":"VAT (ResNet-101)","rank_in_archive_order":51,"of":81,"metrics":{"FB-IoU":"72.4","Mean IoU":"47.9"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-semantic-segmentation-on-fss-1000-1","task":"Few-Shot Semantic Segmentation","dataset":"FSS-1000 (1-shot)","model":"VAT (ResNet-101)","rank_in_archive_order":4,"of":24,"metrics":{"FB-IoU":"94","Mean IoU":"90.3"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-semantic-segmentation-on-fss-1000-1","task":"Few-Shot Semantic Segmentation","dataset":"FSS-1000 (1-shot)","model":"VAT (ResNet-50)","rank_in_archive_order":6,"of":24,"metrics":{"FB-IoU":"93.8","Mean IoU":"90.1"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-semantic-segmentation-on-fss-1000-5","task":"Few-Shot Semantic Segmentation","dataset":"FSS-1000 (5-shot)","model":"VAT (ResNet-101)","rank_in_archive_order":3,"of":22,"metrics":{"FB-IoU":"94.4","Mean IoU":"90.8"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-semantic-segmentation-on-fss-1000-5","task":"Few-Shot Semantic Segmentation","dataset":"FSS-1000 (5-shot)","model":"VAT (ResNet-50)","rank_in_archive_order":4,"of":22,"metrics":{"FB-IoU":"94.2","Mean IoU":"90.7"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-semantic-segmentation-on-pascal-5i-1","task":"Few-Shot Semantic Segmentation","dataset":"PASCAL-5i (1-Shot)","model":"VAT (ResNet-101)","rank_in_archive_order":21,"of":105,"metrics":{"FB-IoU":"79.6","Mean IoU":"67.9"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-semantic-segmentation-on-pascal-5i-1","task":"Few-Shot Semantic Segmentation","dataset":"PASCAL-5i (1-Shot)","model":"VAT (ResNet-50)","rank_in_archive_order":48,"of":105,"metrics":{"FB-IoU":"77.8","Mean IoU":"65.5"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-semantic-segmentation-on-pascal-5i-5","task":"Few-Shot Semantic Segmentation","dataset":"PASCAL-5i (5-Shot)","model":"VAT (ResNet-101)","rank_in_archive_order":19,"of":96,"metrics":{"FB-IoU":"83.2","Mean IoU":"72"},"uses_additional_data":false},{"leaderboard":"/sota/few-shot-semantic-segmentation-on-pascal-5i-5","task":"Few-Shot Semantic Segmentation","dataset":"PASCAL-5i (5-Shot)","model":"VAT (ResNet-50)","rank_in_archive_order":41,"of":96,"metrics":{"FB-IoU":"80.9","Mean IoU":"70.1"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-correspondence-on-pf-pascal","task":"Semantic correspondence","dataset":"PF-PASCAL","model":"VAT (ECCV)","rank_in_archive_order":8,"of":15,"metrics":{"PCK":"92.3"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-correspondence-on-pf-willow","task":"Semantic correspondence","dataset":"PF-WILLOW","model":"VAT (ECCV)","rank_in_archive_order":2,"of":8,"metrics":{"PCK":"81.6"},"uses_additional_data":false},{"leaderboard":"/sota/semantic-correspondence-on-spair-71k","task":"Semantic correspondence","dataset":"SPair-71k","model":"VAT (ECCV)","rank_in_archive_order":12,"of":22,"metrics":{"PCK":"55.5"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2207.10866","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2207.10866"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer","reach":null}],"summary":{"ran":8,"ran_draft_wrong":2,"ran_fixture":1,"unverified":9},"by_repo_kind":{"official":{"samples":18,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"a689dd29d10a5588","entry":"Conv4d","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a689dd29d10a5588"}},{"code_sha256_prefix":"d957466bd907d3e2","entry":"Correlation","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d957466bd907d3e2"}},{"code_sha256_prefix":"97b275801cca78fc","entry":"Interpolate4d","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"97b275801cca78fc"}},{"code_sha256_prefix":"b966405e9e7c94fc","entry":"MaxPool4d","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b966405e9e7c94fc"}},{"code_sha256_prefix":"688a7c109b6a8925","entry":"PatchMerging","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"688a7c109b6a8925"}},{"code_sha256_prefix":"9f47dd1aa3379e7a","entry":"TransformerWarper","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9f47dd1aa3379e7a"}},{"code_sha256_prefix":"3c9c3d947cf636e4","entry":"TransformerWarper2d","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c9c3d947cf636e4"}},{"code_sha256_prefix":"c45ba97aae362041","entry":"WindowAttention","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c45ba97aae362041"}},{"code_sha256_prefix":"cda8eb3e776a0591","entry":"extract_feat_vgg","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cda8eb3e776a0591"}},{"code_sha256_prefix":"f9fd6241d935f07b","entry":"window_partition","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"f9fd6241d935f07b"}},{"code_sha256_prefix":"fb32094c6dbece71","entry":"window_reverse","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"fb32094c6dbece71"}},{"code_sha256_prefix":"30c94d4ce2dcc31b","entry":"BasicLayer","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30c94d4ce2dcc31b"}},{"code_sha256_prefix":"f753e1094cc84a3f","entry":"Encoder4D","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f753e1094cc84a3f"}},{"code_sha256_prefix":"fcf09c7f87136ef1","entry":"OurModel","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fcf09c7f87136ef1"}},{"code_sha256_prefix":"211001493120f0e2","entry":"SwinTransformer","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"211001493120f0e2"}},{"code_sha256_prefix":"4390e9116e6b7b05","entry":"SwinTransformer2d","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4390e9116e6b7b05"}},{"code_sha256_prefix":"cf7fdf758691ee4a","entry":"SwinTransformerBlock","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf7fdf758691ee4a"}},{"code_sha256_prefix":"09ecea11a241565f","entry":"VAT","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"09ecea11a241565f"}},{"code_sha256_prefix":"616c158398e06b3e","entry":"VATLearner","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"616c158398e06b3e"}},{"code_sha256_prefix":"5e71dad5278d69a0","entry":"extract_feat_res","repo":"Seokju-Cho/Volumetric-Aggregation-Transformer","repo_kind":"official","path":"model/vat.py","file_url":"https://github.com/Seokju-Cho/Volumetric-Aggregation-Transformer/blob/HEAD/model/vat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e71dad5278d69a0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}