{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/zero-shot-medical-phrase-grounding-with-off","title":"Zero-Shot Medical Phrase Grounding with Off-the-shelf Diffusion Models","arxiv_id":"2404.12920","date":"2024-04-19","proceeding":null,"authors":["Konstantinos Vilouras","Pedro Sanchez","Alison Q. O'Neil","Sotirios A. Tsaftaris"],"abstract":"Localizing the exact pathological regions in a given medical scan is an important imaging problem that traditionally requires a large amount of bounding box ground truth annotations to be accurately solved. However, there exist alternative, potentially weaker, forms of supervision, such as accompanying free-text reports, which are readily available. The task of performing localization with textual guidance is commonly referred to as phrase grounding. In this work, we use a publicly available Foundation Model, namely the Latent Diffusion Model, to perform this challenging task. This choice is supported by the fact that the Latent Diffusion Model, despite being generative in nature, contains cross-attention mechanisms that implicitly align visual and textual features, thus leading to intermediate representations that are suitable for the task at hand. In addition, we aim to perform this task in a zero-shot manner, i.e., without any training on the target task, meaning that the model's weights remain frozen. To this end, we devise strategies to select features and also refine them via post-processing without extra learnable parameters. We compare our proposed method with state-of-the-art approaches which explicitly enforce image-text alignment in a joint embedding space via contrastive learning. Results on a popular chest X-ray benchmark indicate that our method is competitive with SOTA on different types of pathology, and even outperforms them on average in terms of two metrics (mean IoU and AUC-ROC). Source code will be released upon acceptance at https://github.com/vios-s.","url_abs":"https://arxiv.org/abs/2404.12920v4","url_pdf":"https://arxiv.org/pdf/2404.12920v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"zero-shot-medical-phrase-grounding-with-off","repo_url":"https://github.com/vios-s/ldm-phrase-grounding","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"contrastive-learning","task_name":"Contrastive Learning"},{"task_slug":"phrase-grounding","task_name":"Phrase Grounding"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"diffusion","method_name":"Diffusion"},{"method_slug":"latent-diffusion-model","method_name":"Latent Diffusion Model"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2404.12920","atlas_url":"https://app.syntology.ai/?focus=2404.12920","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.12920"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/vios-s/ldm-phrase-grounding","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":7,"unverified":2},"by_repo_kind":{"official":{"samples":9,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"10140d764aa99bb7","entry":"get_bb_from_largest_component","repo":"vios-s/ldm-phrase-grounding","repo_kind":"official","path":"metrics/utils.py","file_url":"https://github.com/vios-s/ldm-phrase-grounding/blob/HEAD/metrics/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10140d764aa99bb7"}},{"code_sha256_prefix":"0ce3795836af80ec","entry":"get_mask_otsu","repo":"vios-s/ldm-phrase-grounding","repo_kind":"official","path":"sampler.py","file_url":"https://github.com/vios-s/ldm-phrase-grounding/blob/HEAD/sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0ce3795836af80ec"}},{"code_sha256_prefix":"cf6de9501c2993aa","entry":"get_prompt_embeds","repo":"vios-s/ldm-phrase-grounding","repo_kind":"official","path":"models_local/get_models.py","file_url":"https://github.com/vios-s/ldm-phrase-grounding/blob/HEAD/models_local/get_models.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cf6de9501c2993aa"}},{"code_sha256_prefix":"3bd98c6271cef4d2","entry":"largest_connected_component","repo":"vios-s/ldm-phrase-grounding","repo_kind":"official","path":"metrics/utils.py","file_url":"https://github.com/vios-s/ldm-phrase-grounding/blob/HEAD/metrics/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3bd98c6271cef4d2"}},{"code_sha256_prefix":"0529eba846aece7c","entry":"mIoU","repo":"vios-s/ldm-phrase-grounding","repo_kind":"official","path":"metrics/eval_metrics.py","file_url":"https://github.com/vios-s/ldm-phrase-grounding/blob/HEAD/metrics/eval_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0529eba846aece7c"}},{"code_sha256_prefix":"39da679eef535fe1","entry":"nanvar","repo":"vios-s/ldm-phrase-grounding","repo_kind":"official","path":"metrics/utils.py","file_url":"https://github.com/vios-s/ldm-phrase-grounding/blob/HEAD/metrics/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"39da679eef535fe1"}},{"code_sha256_prefix":"ec6fd1be92e74fb3","entry":"normalise_to_01","repo":"vios-s/ldm-phrase-grounding","repo_kind":"official","path":"sampler.py","file_url":"https://github.com/vios-s/ldm-phrase-grounding/blob/HEAD/sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec6fd1be92e74fb3"}},{"code_sha256_prefix":"8fb17b2d83d78dee","entry":"CNR","repo":"vios-s/ldm-phrase-grounding","repo_kind":"official","path":"metrics/eval_metrics.py","file_url":"https://github.com/vios-s/ldm-phrase-grounding/blob/HEAD/metrics/eval_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8fb17b2d83d78dee"}},{"code_sha256_prefix":"6fa294952b71cdb1","entry":"mIoU_old","repo":"vios-s/ldm-phrase-grounding","repo_kind":"official","path":"metrics/eval_metrics.py","file_url":"https://github.com/vios-s/ldm-phrase-grounding/blob/HEAD/metrics/eval_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6fa294952b71cdb1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}