{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/universal-and-extensible-language-vision","title":"Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomography","arxiv_id":"2405.18356","date":"2024-05-28","proceeding":null,"authors":["Jie Liu","Yixiao Zhang","Kang Wang","Mehmet Can Yavuz","Xiaoxi Chen","Yixuan Yuan","Haoliang Li","Yang Yang","Alan Yuille","Yucheng Tang","Zongwei Zhou"],"abstract":"The advancement of artificial intelligence (AI) for organ segmentation and tumor detection is propelled by the growing availability of computed tomography (CT) datasets with detailed, per-voxel annotations. However, these AI models often struggle with flexibility for partially annotated datasets and extensibility for new classes due to limitations in the one-hot encoding, architectural design, and learning scheme. To overcome these limitations, we propose a universal, extensible framework enabling a single model, termed Universal Model, to deal with multiple public datasets and adapt to new classes (e.g., organs/tumors). Firstly, we introduce a novel language-driven parameter generator that leverages language embeddings from large language models, enriching semantic encoding compared with one-hot encoding. Secondly, the conventional output layers are replaced with lightweight, class-specific heads, allowing Universal Model to simultaneously segment 25 organs and six types of tumors and ease the addition of new classes. We train our Universal Model on 3,410 CT volumes assembled from 14 publicly available datasets and then test it on 6,173 CT volumes from four external datasets. Universal Model achieves first place on six CT tasks in the Medical Segmentation Decathlon (MSD) public leaderboard and leading performance on the Beyond The Cranial Vault (BTCV) dataset. In summary, Universal Model exhibits remarkable computational efficiency (6x faster than other dataset-specific models), demonstrates strong generalization across different hospitals, transfers well to numerous downstream tasks, and more importantly, facilitates the extensibility to new classes while alleviating the catastrophic forgetting of previously learned classes. Codes, models, and datasets are available at https://github.com/ljwztc/CLIP-Driven-Universal-Model","url_abs":"https://arxiv.org/abs/2405.18356v1","url_pdf":"https://arxiv.org/pdf/2405.18356v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"universal-and-extensible-language-vision","repo_url":"https://github.com/ljwztc/clip-driven-universal-model","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"},{"task_slug":"computed-tomography-ct","task_name":"Computed Tomography (CT)"},{"task_slug":"organ-segmentation","task_name":"Organ Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.18356","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.18356"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ljwztc/clip-driven-universal-model","reach":{"status":"ok","spdx":"NOASSERTION"}}],"summary":{"ran":1,"ran_fixture":2,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"24feeaf0de319fad","entry":"generate_label","repo":"ljwztc/clip-driven-universal-model","repo_kind":"official","path":"label_transfer.py","file_url":"https://github.com/ljwztc/clip-driven-universal-model/blob/HEAD/label_transfer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"24feeaf0de319fad"}},{"code_sha256_prefix":"168646537c6adfff","entry":"get_window_size","repo":"ljwztc/clip-driven-universal-model","repo_kind":"official","path":"model/SwinUNETR.py","file_url":"https://github.com/ljwztc/clip-driven-universal-model/blob/HEAD/model/SwinUNETR.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"168646537c6adfff"}},{"code_sha256_prefix":"80ada9d6ef99fa0d","entry":"window_partition","repo":"ljwztc/clip-driven-universal-model","repo_kind":"official","path":"model/SwinUNETR.py","file_url":"https://github.com/ljwztc/clip-driven-universal-model/blob/HEAD/model/SwinUNETR.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"80ada9d6ef99fa0d"}},{"code_sha256_prefix":"68a071c415424018","entry":"window_reverse","repo":"ljwztc/clip-driven-universal-model","repo_kind":"official","path":"model/SwinUNETR.py","file_url":"https://github.com/ljwztc/clip-driven-universal-model/blob/HEAD/model/SwinUNETR.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"68a071c415424018"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}