{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/side-adapter-network-for-open-vocabulary","title":"Side Adapter Network for Open-Vocabulary Semantic Segmentation","arxiv_id":"2302.12242","date":"2023-02-23","proceeding":"CVPR 2023 1","authors":["Mengde Xu","Zheng Zhang","Fangyun Wei","Han Hu","Xiang Bai"],"abstract":"This paper presents a new framework for open-vocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN). Our approach models the semantic segmentation task as a region recognition problem. A side network is attached to a frozen CLIP model with two branches: one for predicting mask proposals, and the other for predicting attention bias which is applied in the CLIP model to recognize the class of masks. This decoupled design has the benefit CLIP in recognizing the class of mask proposals. Since the attached side network can reuse CLIP features, it can be very light. In addition, the entire network can be trained end-to-end, allowing the side network to be adapted to the frozen CLIP model, which makes the predicted mask proposals CLIP-aware. Our approach is fast, accurate, and only adds a few additional trainable parameters. We evaluate our approach on multiple semantic segmentation benchmarks. Our method significantly outperforms other counterparts, with up to 18 times fewer trainable parameters and 19 times faster inference speed. We hope our approach will serve as a solid baseline and help ease future research in open-vocabulary semantic segmentation. The code will be available at https://github.com/MendelXu/SAN.","url_abs":"https://arxiv.org/abs/2302.12242v2","url_pdf":"https://arxiv.org/pdf/2302.12242v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"side-adapter-network-for-open-vocabulary","repo_url":"https://github.com/mendelxu/san","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"side-adapter-network-for-open-vocabulary","repo_url":"https://github.com/blumenstiel/SAN-MESS","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"side-adapter-network-for-open-vocabulary","repo_url":"https://github.com/mendelxu/zsseg.baseline","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"open-vocabulary-semantic-segmentation","task_name":"Open Vocabulary Semantic Segmentation"},{"task_slug":"open-vocabulary-semantic-segmentation-1","task_name":"Open-Vocabulary Semantic Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"zero-shot-segmentation","task_name":"Zero Shot Segmentation"}],"methods":[{"method_slug":"adapter","method_name":"Adapter"},{"method_slug":"clip","method_name":"CLIP"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/open-vocabulary-semantic-segmentation-on-3","task":"Open Vocabulary Semantic Segmentation","dataset":"ADE20K-847","model":"SAN","rank_in_archive_order":12,"of":19,"metrics":{"mIoU":"13.7"},"uses_additional_data":false},{"leaderboard":"/sota/zero-shot-segmentation-on-segmentation-in-the","task":"Zero Shot Segmentation","dataset":"Segmentation in the Wild","model":"SAN","rank_in_archive_order":5,"of":12,"metrics":{"Mean AP":"41.4"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2302.12242","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2302.12242"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mendelxu/san","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/blumenstiel/SAN-MESS","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mendelxu/zsseg.baseline","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":1,"unverified":8},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":8,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c266c766b74bafca","entry":"get_predefined_templates","repo":"mendelxu/san","repo_kind":"official","path":"san/model/clip_utils/utils.py","file_url":"https://github.com/mendelxu/san/blob/HEAD/san/model/clip_utils/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c266c766b74bafca"}},{"code_sha256_prefix":"bc2cb481a75c370d","entry":"batch_dice_loss","repo":"blumenstiel/SAN-MESS","repo_kind":"listed","path":"san/model/matcher.py","file_url":"https://github.com/blumenstiel/SAN-MESS/blob/HEAD/san/model/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc2cb481a75c370d"}},{"code_sha256_prefix":"1edd24985036b0bf","entry":"batch_sigmoid_ce_loss","repo":"blumenstiel/SAN-MESS","repo_kind":"listed","path":"san/model/matcher.py","file_url":"https://github.com/blumenstiel/SAN-MESS/blob/HEAD/san/model/matcher.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1edd24985036b0bf"}},{"code_sha256_prefix":"2dcb8123d89bb1ff","entry":"calculate_uncertainty","repo":"blumenstiel/SAN-MESS","repo_kind":"listed","path":"san/model/criterion.py","file_url":"https://github.com/blumenstiel/SAN-MESS/blob/HEAD/san/model/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2dcb8123d89bb1ff"}},{"code_sha256_prefix":"629dc217312c9bb7","entry":"cross_attn_with_self_bias","repo":"blumenstiel/SAN-MESS","repo_kind":"listed","path":"san/model/attn_helper.py","file_url":"https://github.com/blumenstiel/SAN-MESS/blob/HEAD/san/model/attn_helper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"629dc217312c9bb7"}},{"code_sha256_prefix":"920416fe2ccdac57","entry":"cross_attn_with_self_bias_func","repo":"blumenstiel/SAN-MESS","repo_kind":"listed","path":"san/model/attn_helper.py","file_url":"https://github.com/blumenstiel/SAN-MESS/blob/HEAD/san/model/attn_helper.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"920416fe2ccdac57"}},{"code_sha256_prefix":"89f75e54ff128be0","entry":"dice_loss","repo":"blumenstiel/SAN-MESS","repo_kind":"listed","path":"san/model/criterion.py","file_url":"https://github.com/blumenstiel/SAN-MESS/blob/HEAD/san/model/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"89f75e54ff128be0"}},{"code_sha256_prefix":"080cb1a082ca2feb","entry":"get_tiles","repo":"blumenstiel/SAN-MESS","repo_kind":"listed","path":"mess/prepare_datasets/prepare_floodnet.py","file_url":"https://github.com/blumenstiel/SAN-MESS/blob/HEAD/mess/prepare_datasets/prepare_floodnet.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"080cb1a082ca2feb"}},{"code_sha256_prefix":"d0c61e8dba511aa3","entry":"sigmoid_ce_loss","repo":"blumenstiel/SAN-MESS","repo_kind":"listed","path":"san/model/criterion.py","file_url":"https://github.com/blumenstiel/SAN-MESS/blob/HEAD/san/model/criterion.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0c61e8dba511aa3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}