{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/test-time-adaptation-with-salip-a-cascade-of","title":"Test-Time Adaptation with SaLIP: A Cascade of SAM and CLIP for Zero shot Medical Image Segmentation","arxiv_id":"2404.06362","date":"2024-04-09","proceeding":null,"authors":["Sidra Aleem","Fangyijie Wang","Mayug Maniparambil","Eric Arazo","Julia Dietlmeier","Guenole Silvestre","Kathleen Curran","Noel E. O'Connor","Suzanne Little"],"abstract":"The Segment Anything Model (SAM) and CLIP are remarkable vision foundation models (VFMs). SAM, a prompt driven segmentation model, excels in segmentation tasks across diverse domains, while CLIP is renowned for its zero shot recognition capabilities. However, their unified potential has not yet been explored in medical image segmentation. To adapt SAM to medical imaging, existing methods primarily rely on tuning strategies that require extensive data or prior prompts tailored to the specific task, making it particularly challenging when only a limited number of data samples are available. This work presents an in depth exploration of integrating SAM and CLIP into a unified framework for medical image segmentation. Specifically, we propose a simple unified framework, SaLIP, for organ segmentation. Initially, SAM is used for part based segmentation within the image, followed by CLIP to retrieve the mask corresponding to the region of interest (ROI) from the pool of SAM generated masks. Finally, SAM is prompted by the retrieved ROI to segment a specific organ. Thus, SaLIP is training and fine tuning free and does not rely on domain expertise or labeled data for prompt engineering. Our method shows substantial enhancements in zero shot segmentation, showcasing notable improvements in DICE scores across diverse segmentation tasks like brain (63.46%), lung (50.11%), and fetal head (30.82%), when compared to un prompted SAM. Code and text prompts are available at: https://github.com/aleemsidra/SaLIP.","url_abs":"https://arxiv.org/abs/2404.06362v2","url_pdf":"https://arxiv.org/pdf/2404.06362v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"test-time-adaptation-with-salip-a-cascade-of","repo_url":"https://github.com/aleemsidra/SaLIP","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"medical-image-segmentation","task_name":"Medical Image Segmentation"},{"task_slug":"organ-segmentation","task_name":"Organ Segmentation"},{"task_slug":"prompt-engineering","task_name":"Prompt Engineering"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"test-time-adaptation","task_name":"Test-time Adaptation"},{"task_slug":"zero-shot-segmentation","task_name":"Zero Shot Segmentation"},{"task_slug":"zero-shot-learning","task_name":"Zero-Shot Learning"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.06362","atlas_url":"https://app.syntology.ai/?focus=2404.06362","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.06362"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/aleemsidra/SaLIP","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6,"ran_fixture":2,"unverified":4},"by_repo_kind":{"official":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4017666e5cd0801f","entry":"dice_coeff","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"sam_clip_3d.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/sam_clip_3d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4017666e5cd0801f"}},{"code_sha256_prefix":"ce54a742919b8034","entry":"dice_coeff","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ce54a742919b8034"}},{"code_sha256_prefix":"e9b52816a6e8d5b8","entry":"get_crops","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"sam_clip.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/sam_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9b52816a6e8d5b8"}},{"code_sha256_prefix":"733d7f0bedcb74c2","entry":"get_rel_pos","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"segment_anything/modeling/image_encoder.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/segment_anything/modeling/image_encoder.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"733d7f0bedcb74c2"}},{"code_sha256_prefix":"e6af0754224f9b26","entry":"prepare_image","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"sam_clip.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/sam_clip.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e6af0754224f9b26"}},{"code_sha256_prefix":"96dd89ecd1e9afa3","entry":"prepare_image","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"sam_clip_3d.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/sam_clip_3d.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96dd89ecd1e9afa3"}},{"code_sha256_prefix":"07e7154e4c2da99e","entry":"process_config","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"utils/utils.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07e7154e4c2da99e"}},{"code_sha256_prefix":"105fa08885dc36cc","entry":"window_partition","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"segment_anything/modeling/image_encoder.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/segment_anything/modeling/image_encoder.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"105fa08885dc36cc"}},{"code_sha256_prefix":"f9bc5f31ce61cee8","entry":"build_sam_vit_b","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"segment_anything/build_sam.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/segment_anything/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f9bc5f31ce61cee8"}},{"code_sha256_prefix":"6b77c1f11fff3ed6","entry":"build_sam_vit_h","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"segment_anything/build_sam.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/segment_anything/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b77c1f11fff3ed6"}},{"code_sha256_prefix":"3f8890e695469246","entry":"build_sam_vit_l","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"segment_anything/build_sam.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/segment_anything/build_sam.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3f8890e695469246"}},{"code_sha256_prefix":"27be441cc8213e52","entry":"window_unpartition","repo":"aleemsidra/SaLIP","repo_kind":"official","path":"segment_anything/modeling/image_encoder.py","file_url":"https://github.com/aleemsidra/SaLIP/blob/HEAD/segment_anything/modeling/image_encoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"27be441cc8213e52"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}