{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-multimodal-approach-combining-structural","title":"A Multimodal Approach Combining Structural and Cross-domain Textual Guidance for Weakly Supervised OCT Segmentation","arxiv_id":"2411.12615","date":"2024-11-19","proceeding":null,"authors":["Jiaqi Yang","Nitish Mehta","Xiaoling Hu","Chao Chen","Chia-Ling Tsai"],"abstract":"Accurate segmentation of Optical Coherence Tomography (OCT) images is crucial for diagnosing and monitoring retinal diseases. However, the labor-intensive nature of pixel-level annotation limits the scalability of supervised learning with large datasets. Weakly Supervised Semantic Segmentation (WSSS) provides a promising alternative by leveraging image-level labels. In this study, we propose a novel WSSS approach that integrates structural guidance with text-driven strategies to generate high-quality pseudo labels, significantly improving segmentation performance. In terms of visual information, our method employs two processing modules that exchange raw image features and structural features from OCT images, guiding the model to identify where lesions are likely to occur. In terms of textual information, we utilize large-scale pretrained models from cross-domain sources to implement label-informed textual guidance and synthetic descriptive integration with two textual processing modules that combine local semantic features with consistent synthetic descriptions. By fusing these visual and textual components within a multimodal framework, our approach enhances lesion localization accuracy. Experimental results on three OCT datasets demonstrate that our method achieves state-of-the-art performance, highlighting its potential to improve diagnostic accuracy and efficiency in medical imaging.","url_abs":"https://arxiv.org/abs/2411.12615v1","url_pdf":"https://arxiv.org/pdf/2411.12615v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-multimodal-approach-combining-structural","repo_url":"https://github.com/yangjiaqidig/WSSS-AGM","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"descriptive","task_name":"Descriptive"},{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"weakly-supervised-semantic-segmentation-1","task_name":"Weakly supervised Semantic Segmentation"},{"task_slug":"weakly-supervised-semantic-segmentation","task_name":"Weakly-Supervised Semantic Segmentation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2411.12615","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.12615"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yangjiaqidig/WSSS-AGM","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"unverified":6},"by_repo_kind":{"official":{"samples":8,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"yangjiaqidig/WSSS-AGM","repo_kind":"official","path":"anomaly_guided/alternatives_network/baselines.py","file_url":"https://github.com/yangjiaqidig/WSSS-AGM/blob/HEAD/anomaly_guided/alternatives_network/baselines.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"160bb14bd76201b4","entry":"conv3x3","repo":"yangjiaqidig/WSSS-AGM","repo_kind":"official","path":"anomaly_guided/alternatives_network/baselines.py","file_url":"https://github.com/yangjiaqidig/WSSS-AGM/blob/HEAD/anomaly_guided/alternatives_network/baselines.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"160bb14bd76201b4"}},{"code_sha256_prefix":"9b7b7113c6561851","entry":"diff_map_for_att","repo":"yangjiaqidig/WSSS-AGM","repo_kind":"official","path":"structure_analysis.py","file_url":"https://github.com/yangjiaqidig/WSSS-AGM/blob/HEAD/structure_analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9b7b7113c6561851"}},{"code_sha256_prefix":"e7ea1678b8cc6ef1","entry":"gan_normalize_transform","repo":"yangjiaqidig/WSSS-AGM","repo_kind":"official","path":"anomaly_guided/dataset.py","file_url":"https://github.com/yangjiaqidig/WSSS-AGM/blob/HEAD/anomaly_guided/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e7ea1678b8cc6ef1"}},{"code_sha256_prefix":"803b34cebd1c61cc","entry":"get_gan_path_by_image_path","repo":"yangjiaqidig/WSSS-AGM","repo_kind":"official","path":"structure_analysis.py","file_url":"https://github.com/yangjiaqidig/WSSS-AGM/blob/HEAD/structure_analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"803b34cebd1c61cc"}},{"code_sha256_prefix":"ddf12fd4850c33a7","entry":"img_transform","repo":"yangjiaqidig/WSSS-AGM","repo_kind":"official","path":"anomaly_guided/dataset.py","file_url":"https://github.com/yangjiaqidig/WSSS-AGM/blob/HEAD/anomaly_guided/dataset.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ddf12fd4850c33a7"}},{"code_sha256_prefix":"1b8f25b625b05b10","entry":"network_class","repo":"yangjiaqidig/WSSS-AGM","repo_kind":"official","path":"anomaly_guided/network/utils.py","file_url":"https://github.com/yangjiaqidig/WSSS-AGM/blob/HEAD/anomaly_guided/network/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1b8f25b625b05b10"}},{"code_sha256_prefix":"09145a832d198e3a","entry":"normalized_batch_tensor","repo":"yangjiaqidig/WSSS-AGM","repo_kind":"official","path":"structure_analysis.py","file_url":"https://github.com/yangjiaqidig/WSSS-AGM/blob/HEAD/structure_analysis.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09145a832d198e3a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}