{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/blo-sam-bi-level-optimization-based","title":"BLO-SAM: Bi-level Optimization Based Overfitting-Preventing Finetuning of SAM","arxiv_id":"2402.16338","date":"2024-02-26","proceeding":null,"authors":["Li Zhang","Youwei Liang","Ruiyi Zhang","Amirhosein Javadi","Pengtao Xie"],"abstract":"The Segment Anything Model (SAM), a foundation model pretrained on millions of images and segmentation masks, has significantly advanced semantic segmentation, a fundamental task in computer vision. Despite its strengths, SAM encounters two major challenges. Firstly, it struggles with segmenting specific objects autonomously, as it relies on users to manually input prompts like points or bounding boxes to identify targeted objects. Secondly, SAM faces challenges in excelling at specific downstream tasks, like medical imaging, due to a disparity between the distribution of its pretraining data, which predominantly consists of general-domain images, and the data used in downstream tasks. Current solutions to these problems, which involve finetuning SAM, often lead to overfitting, a notable issue in scenarios with very limited data, like in medical imaging. To overcome these limitations, we introduce BLO-SAM, which finetunes SAM based on bi-level optimization (BLO). Our approach allows for automatic image segmentation without the need for manual prompts, by optimizing a learnable prompt embedding. Furthermore, it significantly reduces the risk of overfitting by training the model's weight parameters and the prompt embedding on two separate subsets of the training dataset, each at a different level of optimization. We apply BLO-SAM to diverse semantic segmentation tasks in general and medical domains. The results demonstrate BLO-SAM's superior performance over various state-of-the-art image semantic segmentation methods.","url_abs":"https://arxiv.org/abs/2402.16338v4","url_pdf":"https://arxiv.org/pdf/2402.16338v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"blo-sam-bi-level-optimization-based","repo_url":"https://github.com/importzl/blo-sam","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"}],"methods":[{"method_slug":"sam","method_name":"SAM"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.16338","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.16338"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/importzl/blo-sam","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":4},"by_repo_kind":{"official":{"samples":8,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"ad5471db7ab93cbf","entry":"calc_loss","repo":"importzl/blo-sam","repo_kind":"official","path":"trainer.py","file_url":"https://github.com/importzl/blo-sam/blob/HEAD/trainer.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad5471db7ab93cbf"}},{"code_sha256_prefix":"5c939eca7a512176","entry":"calc_loss","repo":"importzl/blo-sam","repo_kind":"official","path":"prompt.py","file_url":"https://github.com/importzl/blo-sam/blob/HEAD/prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c939eca7a512176"}},{"code_sha256_prefix":"c760bb97db1f5079","entry":"config_to_dict","repo":"importzl/blo-sam","repo_kind":"official","path":"inference.py","file_url":"https://github.com/importzl/blo-sam/blob/HEAD/inference.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c760bb97db1f5079"}},{"code_sha256_prefix":"5b1f6d6ac23c60e7","entry":"dice_score","repo":"importzl/blo-sam","repo_kind":"official","path":"cal_dice.py","file_url":"https://github.com/importzl/blo-sam/blob/HEAD/cal_dice.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5b1f6d6ac23c60e7"}},{"code_sha256_prefix":"d989416cca0adba7","entry":"build_sam_vit_b","repo":"importzl/blo-sam","repo_kind":"official","path":"segment_anything/build_sam.py","file_url":"https://github.com/importzl/blo-sam/blob/HEAD/segment_anything/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d989416cca0adba7"}},{"code_sha256_prefix":"4b02e5ca19e6ccd4","entry":"build_sam_vit_h","repo":"importzl/blo-sam","repo_kind":"official","path":"segment_anything/build_sam.py","file_url":"https://github.com/importzl/blo-sam/blob/HEAD/segment_anything/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4b02e5ca19e6ccd4"}},{"code_sha256_prefix":"652f2a87b8537ea1","entry":"build_sam_vit_l","repo":"importzl/blo-sam","repo_kind":"official","path":"segment_anything/build_sam.py","file_url":"https://github.com/importzl/blo-sam/blob/HEAD/segment_anything/build_sam.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"652f2a87b8537ea1"}},{"code_sha256_prefix":"3b1ea92c98eff298","entry":"getStat","repo":"importzl/blo-sam","repo_kind":"official","path":"utils.py","file_url":"https://github.com/importzl/blo-sam/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b1ea92c98eff298"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}