{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/one-model-to-rule-them-all-towards-universal","title":"One Model to Rule them All: Towards Universal Segmentation for Medical Images with Text Prompts","arxiv_id":"2312.17183","date":"2023-12-28","proceeding":null,"authors":["Ziheng Zhao","Yao Zhang","Chaoyi Wu","Xiaoman Zhang","Ya zhang","Yanfeng Wang","Weidi Xie"],"abstract":"In this study, we aim to build up a model that can Segment Anything in radiology scans, driven by medical terminologies as Text prompts, termed as SAT. Our main contributions are three folds: (i) for dataset construction, we construct the first multi-modal knowledge tree on human anatomy, including 6502 anatomical terminologies; Then, we build up the largest and most comprehensive segmentation dataset for training, by collecting over 22K 3D medical image scans from72 segmentation datasets, across 497 classes, with careful standardization on both image scans and label space; (ii) for architecture design, we propose to inject medical knowledge into a text encoder via contrastive learning, and then formulate a universal segmentation model, that can be prompted by feeding in medical terminologies in text form; (iii) As a result, we have trained SAT-Nano (110M parameters) and SAT-Pro (447M parameters), demonstrating superior or comparable performance to 72 specialist models, i.e., nnU-Nets, U-Mamba or SwinUNETR, trained on each dataset/subsets. We validate SAT as a foundational segmentation model, with better generalization on external (cross-center) datasets, and can be further improved on specific tasks after fine-tuning adaptation. Comparing with state-of-the-art interactive segmentation model MedSAM, SAT demonstrate superior performance, scalability and robustness. We further compare SAT with BiomedParse, and observe SAT is significantly superior in both internal and external evaluation. Through extensive ablation study, we validate the benefit of domain knowledge on universal segmentation, especially on tail categories. As a use case, we demonstrate that SAT can act as a powerful out-of-the-box agent for large language models, enabling visual grounding in versatile application scenarios. All the data, codes, and models in this work have been released.","url_abs":"https://arxiv.org/abs/2312.17183v4","url_pdf":"https://arxiv.org/pdf/2312.17183v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"one-model-to-rule-them-all-towards-universal","repo_url":"https://github.com/zhaoziheng/sat-ds","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"one-model-to-rule-them-all-towards-universal","repo_url":"https://github.com/zhaoziheng/sat","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"anatomy","task_name":"Anatomy"},{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"interactive-segmentation","task_name":"Interactive Segmentation"},{"task_slug":"mamba","task_name":"Mamba"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"universal-segmentation","task_name":"Universal Segmentation"},{"task_slug":"visual-grounding","task_name":"Visual Grounding"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2312.17183","atlas_url":"https://app.syntology.ai/?focus=2312.17183","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.17183"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaoziheng/sat","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhaoziheng/sat-ds","reach":{"status":"ok"}}],"summary":{"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"a6bc7496b6b250f9","entry":"Normalization","repo":"zhaoziheng/sat-ds","repo_kind":"official","path":"loader.py","file_url":"https://github.com/zhaoziheng/sat-ds/blob/HEAD/loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a6bc7496b6b250f9"}},{"code_sha256_prefix":"a518e29963d7ed31","entry":"contains","repo":"zhaoziheng/sat-ds","repo_kind":"official","path":"train_test_split.py","file_url":"https://github.com/zhaoziheng/sat-ds/blob/HEAD/train_test_split.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a518e29963d7ed31"}},{"code_sha256_prefix":"321b45a28a2354f6","entry":"npy_loader","repo":"zhaoziheng/sat-ds","repo_kind":"official","path":"loader_npy.py","file_url":"https://github.com/zhaoziheng/sat-ds/blob/HEAD/loader_npy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"321b45a28a2354f6"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}