{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/gsva-generalized-segmentation-via-multimodal","title":"GSVA: Generalized Segmentation via Multimodal Large Language Models","arxiv_id":"2312.10103","date":"2023-12-15","proceeding":"CVPR 2024 1","authors":["Zhuofan Xia","Dongchen Han","Yizeng Han","Xuran Pan","Shiji Song","Gao Huang"],"abstract":"Generalized Referring Expression Segmentation (GRES) extends the scope of classic RES to refer to multiple objects in one expression or identify the empty targets absent in the image. GRES poses challenges in modeling the complex spatial relationships of the instances in the image and identifying non-existing referents. Multimodal Large Language Models (MLLMs) have recently shown tremendous progress in these complicated vision-language tasks. Connecting Large Language Models (LLMs) and vision models, MLLMs are proficient in understanding contexts with visual inputs. Among them, LISA, as a representative, adopts a special [SEG] token to prompt a segmentation mask decoder, e.g., SAM, to enable MLLMs in the RES task. However, existing solutions to GRES remain unsatisfactory since current segmentation MLLMs cannot correctly handle the cases where users might reference multiple subjects in a singular prompt or provide descriptions incongruent with any image target. In this paper, we propose Generalized Segmentation Vision Assistant (GSVA) to address this gap. Specifically, GSVA reuses the [SEG] token to prompt the segmentation model towards supporting multiple mask references simultaneously and innovatively learns to generate a [REJ] token to reject the null targets explicitly. Experiments validate GSVA's efficacy in resolving the GRES issue, marking a notable enhancement and setting a new record on the GRES benchmark gRefCOCO dataset. GSVA also proves effective across various classic referring segmentation and comprehension tasks.","url_abs":"https://arxiv.org/abs/2312.10103v3","url_pdf":"https://arxiv.org/pdf/2312.10103v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"gsva-generalized-segmentation-via-multimodal","repo_url":"https://github.com/leaplabthu/gsva","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"generalized-referring-expression-segmentation","task_name":"Generalized Referring Expression Segmentation"},{"task_slug":"referring-expression","task_name":"Referring Expression"},{"task_slug":"referring-expression-segmentation","task_name":"Referring Expression Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"}],"methods":[{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/generalized-referring-expression-segmentation","task":"Generalized Referring Expression Segmentation","dataset":"gRefCOCO","model":"GSVA-Llama2-13B","rank_in_archive_order":2,"of":13,"metrics":{"cIoU":"66.38","gIoU":"70.04"},"uses_additional_data":true},{"leaderboard":"/sota/generalized-referring-expression-segmentation","task":"Generalized Referring Expression Segmentation","dataset":"gRefCOCO","model":"GSVA-Vicuna-13B-v1.1","rank_in_archive_order":5,"of":13,"metrics":{"cIoU":"64.05","gIoU":"68.01"},"uses_additional_data":true},{"leaderboard":"/sota/generalized-referring-expression-segmentation","task":"Generalized Referring Expression Segmentation","dataset":"gRefCOCO","model":"GSVA-Vicuna-7B-v1.1","rank_in_archive_order":7,"of":13,"metrics":{"cIoU":"63.29","gIoU":"66.47"},"uses_additional_data":true}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.10103","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.10103"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/leaplabthu/gsva","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":2,"unverified":8},"by_repo_kind":{"official":{"samples":10,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7e11d9ec27335f5b","entry":"get_logger","repo":"leaplabthu/gsva","repo_kind":"official","path":"utils/logger.py","file_url":"https://github.com/leaplabthu/gsva/blob/HEAD/utils/logger.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7e11d9ec27335f5b"}},{"code_sha256_prefix":"1df990c375318896","entry":"process_images","repo":"leaplabthu/gsva","repo_kind":"official","path":"model/llava/mm_utils.py","file_url":"https://github.com/leaplabthu/gsva/blob/HEAD/model/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1df990c375318896"}},{"code_sha256_prefix":"e11c7f60d91192ce","entry":"build_sam_vit_b","repo":"leaplabthu/gsva","repo_kind":"official","path":"model/segment_anything/build_sam.py","file_url":"https://github.com/leaplabthu/gsva/blob/HEAD/model/segment_anything/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e11c7f60d91192ce"}},{"code_sha256_prefix":"06cfc37f79036a2c","entry":"build_sam_vit_h","repo":"leaplabthu/gsva","repo_kind":"official","path":"model/segment_anything/build_sam.py","file_url":"https://github.com/leaplabthu/gsva/blob/HEAD/model/segment_anything/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"06cfc37f79036a2c"}},{"code_sha256_prefix":"7cea226e4ef7b90d","entry":"build_sam_vit_l","repo":"leaplabthu/gsva","repo_kind":"official","path":"model/segment_anything/build_sam.py","file_url":"https://github.com/leaplabthu/gsva/blob/HEAD/model/segment_anything/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7cea226e4ef7b90d"}},{"code_sha256_prefix":"6ec47ecac3fe27fc","entry":"dice_loss","repo":"leaplabthu/gsva","repo_kind":"official","path":"model/losses.py","file_url":"https://github.com/leaplabthu/gsva/blob/HEAD/model/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6ec47ecac3fe27fc"}},{"code_sha256_prefix":"0a8545604f9b15d8","entry":"init_vision_seg_for_model","repo":"leaplabthu/gsva","repo_kind":"official","path":"model/prepare_model_tokenizer.py","file_url":"https://github.com/leaplabthu/gsva/blob/HEAD/model/prepare_model_tokenizer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0a8545604f9b15d8"}},{"code_sha256_prefix":"af6fc3d42a9de09a","entry":"iou_loss","repo":"leaplabthu/gsva","repo_kind":"official","path":"model/losses.py","file_url":"https://github.com/leaplabthu/gsva/blob/HEAD/model/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"af6fc3d42a9de09a"}},{"code_sha256_prefix":"0dde2e782959c0bd","entry":"load_image_from_base64","repo":"leaplabthu/gsva","repo_kind":"official","path":"model/llava/mm_utils.py","file_url":"https://github.com/leaplabthu/gsva/blob/HEAD/model/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0dde2e782959c0bd"}},{"code_sha256_prefix":"13c0bb91f8009d9c","entry":"sigmoid_ce_loss","repo":"leaplabthu/gsva","repo_kind":"official","path":"model/losses.py","file_url":"https://github.com/leaplabthu/gsva/blob/HEAD/model/losses.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"13c0bb91f8009d9c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}