{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lasagna-language-based-segmentation-assistant","title":"LaSagnA: Language-based Segmentation Assistant for Complex Queries","arxiv_id":"2404.08506","date":"2024-04-12","proceeding":null,"authors":["Cong Wei","Haoxian Tan","Yujie Zhong","Yujiu Yang","Lin Ma"],"abstract":"Recent advancements have empowered Large Language Models for Vision (vLLMs) to generate detailed perceptual outcomes, including bounding boxes and masks. Nonetheless, there are two constraints that restrict the further application of these vLLMs: the incapability of handling multiple targets per query and the failure to identify the absence of query objects in the image. In this study, we acknowledge that the main cause of these problems is the insufficient complexity of training queries. Consequently, we define the general sequence format for complex queries. Then we incorporate a semantic segmentation task in the current pipeline to fulfill the requirements of training data. Furthermore, we present three novel strategies to effectively handle the challenges arising from the direct integration of the proposed format. The effectiveness of our model in processing complex queries is validated by the comparable results with conventional methods on both close-set and open-set semantic segmentation datasets. Additionally, we outperform a series of vLLMs in reasoning and referring segmentation, showcasing our model's remarkable capabilities. We release the code at https://github.com/congvvc/LaSagnA.","url_abs":"https://arxiv.org/abs/2404.08506v1","url_pdf":"https://arxiv.org/pdf/2404.08506v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lasagna-language-based-segmentation-assistant","repo_url":"https://github.com/congvvc/lasagna","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2404.08506","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.08506"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/congvvc/lasagna","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_honours":3,"ran":4,"ran_fixture":3,"unverified":6},"by_repo_kind":{"official":{"samples":16,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e0a06ded5d4f6c3c","entry":"box_cxcywh_to_xyxy","repo":"congvvc/lasagna","repo_kind":"official","path":"model/matcher.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/matcher.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e0a06ded5d4f6c3c"}},{"code_sha256_prefix":"ce749f837424dd8c","entry":"box_iou","repo":"congvvc/lasagna","repo_kind":"official","path":"model/matcher.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/matcher.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce749f837424dd8c"}},{"code_sha256_prefix":"34b5a8556963145a","entry":"box_xywh_to_cxcywh","repo":"congvvc/lasagna","repo_kind":"official","path":"utils/box_ops.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/utils/box_ops.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34b5a8556963145a"}},{"code_sha256_prefix":"031b02fb990284ba","entry":"box_xyxy_to_cxcywh","repo":"congvvc/lasagna","repo_kind":"official","path":"utils/box_ops.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/utils/box_ops.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"031b02fb990284ba"}},{"code_sha256_prefix":"a1c2083ec2eaeb22","entry":"dice_loss","repo":"congvvc/lasagna","repo_kind":"official","path":"model/LaSagnA.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/LaSagnA.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a1c2083ec2eaeb22"}},{"code_sha256_prefix":"dc8fb9b6fb48ac37","entry":"generalized_box_iou","repo":"congvvc/lasagna","repo_kind":"official","path":"model/matcher.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/matcher.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dc8fb9b6fb48ac37"}},{"code_sha256_prefix":"dfaf6628e326c423","entry":"list_of_str","repo":"congvvc/lasagna","repo_kind":"official","path":"train_ds.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/train_ds.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"dfaf6628e326c423"}},{"code_sha256_prefix":"52a717cc8a70608b","entry":"parse_args","repo":"congvvc/lasagna","repo_kind":"official","path":"merge_lora_weights_and_save_hf_model.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/merge_lora_weights_and_save_hf_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"52a717cc8a70608b"}},{"code_sha256_prefix":"1df990c375318896","entry":"process_images","repo":"congvvc/lasagna","repo_kind":"official","path":"model/llava/mm_utils.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1df990c375318896"}},{"code_sha256_prefix":"a9292f5d89194794","entry":"sigmoid_ce_loss","repo":"congvvc/lasagna","repo_kind":"official","path":"model/LaSagnA.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/LaSagnA.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a9292f5d89194794"}},{"code_sha256_prefix":"3f50fb8374909f5f","entry":"build_sam_vit_b","repo":"congvvc/lasagna","repo_kind":"official","path":"model/segment_anything/build_sam.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/segment_anything/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3f50fb8374909f5f"}},{"code_sha256_prefix":"bf61a91555ac5280","entry":"build_sam_vit_h","repo":"congvvc/lasagna","repo_kind":"official","path":"model/segment_anything/build_sam.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/segment_anything/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf61a91555ac5280"}},{"code_sha256_prefix":"e2ae4c696b8c968c","entry":"build_sam_vit_l","repo":"congvvc/lasagna","repo_kind":"official","path":"model/segment_anything/build_sam.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/segment_anything/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e2ae4c696b8c968c"}},{"code_sha256_prefix":"0dde2e782959c0bd","entry":"load_image_from_base64","repo":"congvvc/lasagna","repo_kind":"official","path":"model/llava/mm_utils.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/llava/mm_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0dde2e782959c0bd"}},{"code_sha256_prefix":"37899f22fb191b37","entry":"pretty_print_semaphore","repo":"congvvc/lasagna","repo_kind":"official","path":"model/llava/utils.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37899f22fb191b37"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"congvvc/lasagna","repo_kind":"official","path":"model/llava/utils.py","file_url":"https://github.com/congvvc/lasagna/blob/HEAD/model/llava/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}