{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decomposing-disease-descriptions-for-enhanced","title":"Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework","arxiv_id":"2403.07636","date":"2024-03-12","proceeding":"CVPR 2024 1","authors":["Vu Minh Hieu Phan","Yutong Xie","Yuankai Qi","Lingqiao Liu","Liyang Liu","BoWen Zhang","Zhibin Liao","Qi Wu","Minh-Son To","Johan W. Verjans"],"abstract":"Medical vision language pre-training (VLP) has emerged as a frontier of research, enabling zero-shot pathological recognition by comparing the query image with the textual descriptions for each disease. Due to the complex semantics of biomedical texts, current methods struggle to align medical images with key pathological findings in unstructured reports. This leads to the misalignment with the target disease's textual representation. In this paper, we introduce a novel VLP framework designed to dissect disease descriptions into their fundamental aspects, leveraging prior knowledge about the visual manifestations of pathologies. This is achieved by consulting a large language model and medical experts. Integrating a Transformer module, our approach aligns an input image with the diverse elements of a disease, generating aspect-centric image representations. By consolidating the matches from each aspect, we improve the compatibility between an image and its associated disease. Additionally, capitalizing on the aspect-oriented representations, we present a dual-head Transformer tailored to process known and unknown diseases, optimizing the comprehensive detection efficacy. Conducting experiments on seven downstream datasets, ours improves the accuracy of recent methods by up to 8.56% and 17.26% for seen and unseen categories, respectively. Our code is released at https://github.com/HieuPhan33/MAVL.","url_abs":"https://arxiv.org/abs/2403.07636v4","url_pdf":"https://arxiv.org/pdf/2403.07636v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decomposing-disease-descriptions-for-enhanced","repo_url":"https://github.com/hieuphan33/mavl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"decomposing-disease-descriptions-for-enhanced","repo_url":"https://github.com/hieuphan33/cvpr2024_mavl","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[{"method_slug":"align","method_name":"ALIGN"},{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2403.07636","atlas_url":"https://app.syntology.ai/?focus=2403.07636","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.07636"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hieuphan33/cvpr2024_mavl","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hieuphan33/mavl","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/HieuPhan33/MAVL","reach":{"status":"ok"}}],"summary":{"ran":9,"ran_fixture":1,"ran_draft_wrong":1,"unverified":2},"by_repo_kind":{"official":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"989ccd27e10e989a","entry":"AttentionPool2d","repo":"HieuPhan33/MAVL","repo_kind":"official","path":"Pretrain/models/model_MAVL.py","file_url":"https://github.com/HieuPhan33/MAVL/blob/HEAD/Pretrain/models/model_MAVL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"989ccd27e10e989a"}},{"code_sha256_prefix":"5c4a9f9e4ae8ecc2","entry":"SoftTargetCrossEntropy","repo":"HieuPhan33/MAVL","repo_kind":"official","path":"Pretrain/models/model_MAVL.py","file_url":"https://github.com/HieuPhan33/MAVL/blob/HEAD/Pretrain/models/model_MAVL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5c4a9f9e4ae8ecc2"}},{"code_sha256_prefix":"3f7d4237b7f3b04b","entry":"compute_AUCs","repo":"hieuphan33/mavl","repo_kind":"official","path":"Finetuning/classification/train_res_ft.py","file_url":"https://github.com/hieuphan33/mavl/blob/HEAD/Finetuning/classification/train_res_ft.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3f7d4237b7f3b04b"}},{"code_sha256_prefix":"aa81d03f07735ab8","entry":"compute_acc","repo":"hieuphan33/mavl","repo_kind":"official","path":"Finetuning/classification/utils.py","file_url":"https://github.com/hieuphan33/mavl/blob/HEAD/Finetuning/classification/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa81d03f07735ab8"}},{"code_sha256_prefix":"ad12a494674d23fb","entry":"compute_n_params","repo":"hieuphan33/mavl","repo_kind":"official","path":"Finetuning/classification/utils.py","file_url":"https://github.com/hieuphan33/mavl/blob/HEAD/Finetuning/classification/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad12a494674d23fb"}},{"code_sha256_prefix":"b1c51e59891b9aa9","entry":"f2_score","repo":"hieuphan33/mavl","repo_kind":"official","path":"Finetuning/segmentation/metric.py","file_url":"https://github.com/hieuphan33/mavl/blob/HEAD/Finetuning/segmentation/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1c51e59891b9aa9"}},{"code_sha256_prefix":"fbc8bc969c0d3239","entry":"get_tokenizer","repo":"hieuphan33/mavl","repo_kind":"official","path":"Pretrain/train_MAVL.py","file_url":"https://github.com/hieuphan33/mavl/blob/HEAD/Pretrain/train_MAVL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fbc8bc969c0d3239"}},{"code_sha256_prefix":"63fb84685fadf012","entry":"mIoU","repo":"hieuphan33/mavl","repo_kind":"official","path":"Finetuning/segmentation/metric.py","file_url":"https://github.com/hieuphan33/mavl/blob/HEAD/Finetuning/segmentation/metric.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"63fb84685fadf012"}},{"code_sha256_prefix":"0d00e605b1ffaceb","entry":"pairwise_loss","repo":"hieuphan33/mavl","repo_kind":"official","path":"Pretrain/models/loss.py","file_url":"https://github.com/hieuphan33/mavl/blob/HEAD/Pretrain/models/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0d00e605b1ffaceb"}},{"code_sha256_prefix":"0f711eef730b6b3d","entry":"train","repo":"hieuphan33/mavl","repo_kind":"official","path":"Pretrain/train_MAVL.py","file_url":"https://github.com/hieuphan33/mavl/blob/HEAD/Pretrain/train_MAVL.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0f711eef730b6b3d"}},{"code_sha256_prefix":"00bdf7b6512e210e","entry":"valid","repo":"hieuphan33/mavl","repo_kind":"official","path":"Finetuning/classification/train_res_ft.py","file_url":"https://github.com/hieuphan33/mavl/blob/HEAD/Finetuning/classification/train_res_ft.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"00bdf7b6512e210e"}},{"code_sha256_prefix":"6e1b9e01f910b4b2","entry":"MAVL","repo":"HieuPhan33/MAVL","repo_kind":"official","path":"Pretrain/models/model_MAVL.py","file_url":"https://github.com/HieuPhan33/MAVL/blob/HEAD/Pretrain/models/model_MAVL.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6e1b9e01f910b4b2"}},{"code_sha256_prefix":"6d96a71ad7652412","entry":"dice","repo":"hieuphan33/mavl","repo_kind":"official","path":"Finetuning/segmentation/metric.py","file_url":"https://github.com/hieuphan33/mavl/blob/HEAD/Finetuning/segmentation/metric.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6d96a71ad7652412"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}