{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mmed-rag-versatile-multimodal-rag-system-for","title":"MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models","arxiv_id":"2410.13085","date":"2024-10-16","proceeding":null,"authors":["Peng Xia","Kangyu Zhu","Haoran Li","Tianze Wang","Weijia Shi","Sheng Wang","Linjun Zhang","James Zou","Huaxiu Yao"],"abstract":"Artificial Intelligence (AI) has demonstrated significant potential in healthcare, particularly in disease diagnosis and treatment planning. Recent progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new possibilities for interactive diagnostic tools. However, these models often suffer from factual hallucination, which can lead to incorrect diagnoses. Fine-tuning and retrieval-augmented generation (RAG) have emerged as methods to address these issues. However, the amount of high-quality data and distribution shifts between training data and deployment data limit the application of fine-tuning methods. Although RAG is lightweight and effective, existing RAG-based approaches are not sufficiently general to different medical domains and can potentially cause misalignment issues, both between modalities and between the model and the ground truth. In this paper, we propose a versatile multimodal RAG system, MMed-RAG, designed to enhance the factuality of Med-LVLMs. Our approach introduces a domain-aware retrieval mechanism, an adaptive retrieved contexts selection method, and a provable RAG-based preference fine-tuning strategy. These innovations make the RAG process sufficiently general and reliable, significantly improving alignment when introducing retrieved contexts. Experimental results across five medical datasets (involving radiology, ophthalmology, pathology) on medical VQA and report generation demonstrate that MMed-RAG can achieve an average improvement of 43.8% in the factual accuracy of Med-LVLMs. Our data and code are available in https://github.com/richard-peng-xia/MMed-RAG.","url_abs":"https://arxiv.org/abs/2410.13085v1","url_pdf":"https://arxiv.org/pdf/2410.13085v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mmed-rag-versatile-multimodal-rag-system-for","repo_url":"https://github.com/richard-peng-xia/mmed-rag","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"diagnostic","task_name":"Diagnostic"},{"task_slug":"hallucination","task_name":"Hallucination"},{"task_slug":"rag","task_name":"RAG"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bart","method_name":"BART"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"rag","method_name":"RAG"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.13085","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.13085"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/richard-peng-xia/MMed-RAG","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/richard-peng-xia/mmed-rag","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":2,"ran_draft_wrong":2,"ran_honours":1,"ran_fixture":2,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"3477a2c3e334616b","entry":"clean_data_info","repo":"richard-peng-xia/MMed-RAG","repo_kind":"official","path":"train/open_clip/src/retrieve_clip_VQA.py","file_url":"https://github.com/richard-peng-xia/MMed-RAG/blob/HEAD/train/open_clip/src/retrieve_clip_VQA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3477a2c3e334616b"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"richard-peng-xia/MMed-RAG","repo_kind":"official","path":"train/dpo/povid_infer.py","file_url":"https://github.com/richard-peng-xia/MMed-RAG/blob/HEAD/train/dpo/povid_infer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"a8c80549ff61c7ea","entry":"get_logits","repo":"richard-peng-xia/MMed-RAG","repo_kind":"official","path":"train/open_clip/src/retrieve_clip_VQA.py","file_url":"https://github.com/richard-peng-xia/MMed-RAG/blob/HEAD/train/open_clip/src/retrieve_clip_VQA.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8c80549ff61c7ea"}},{"code_sha256_prefix":"9b3c1cb391672ccb","entry":"load_image","repo":"richard-peng-xia/MMed-RAG","repo_kind":"official","path":"train/dpo/predict.py","file_url":"https://github.com/richard-peng-xia/MMed-RAG/blob/HEAD/train/dpo/predict.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9b3c1cb391672ccb"}},{"code_sha256_prefix":"f89c524e425ca547","entry":"retrieve_topk_per_image","repo":"richard-peng-xia/MMed-RAG","repo_kind":"official","path":"train/open_clip/src/retrieve_clip_VQA.py","file_url":"https://github.com/richard-peng-xia/MMed-RAG/blob/HEAD/train/open_clip/src/retrieve_clip_VQA.py","link_basis":"plan_row","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f89c524e425ca547"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"richard-peng-xia/MMed-RAG","repo_kind":"official","path":"train/dpo/povid_infer.py","file_url":"https://github.com/richard-peng-xia/MMed-RAG/blob/HEAD/train/dpo/povid_infer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"10893c4608c08075","entry":"split_to_even_chunks","repo":"richard-peng-xia/MMed-RAG","repo_kind":"official","path":"train/dpo/llava_trainer_2stages.py","file_url":"https://github.com/richard-peng-xia/MMed-RAG/blob/HEAD/train/dpo/llava_trainer_2stages.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"10893c4608c08075"}},{"code_sha256_prefix":"b9978345e2dba578","entry":"LlavaMetaForCausalLM","repo":"richard-peng-xia/mmed-rag","repo_kind":"official","path":"train/dpo/llava/model/llava_arch.py","file_url":"https://github.com/richard-peng-xia/mmed-rag/blob/HEAD/train/dpo/llava/model/llava_arch.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9978345e2dba578"}},{"code_sha256_prefix":"bb35e3ac741bb2c9","entry":"get_mm_adapter_state_maybe_zero_3","repo":"richard-peng-xia/MMed-RAG","repo_kind":"official","path":"train/dpo/llava_trainer_2stages.py","file_url":"https://github.com/richard-peng-xia/MMed-RAG/blob/HEAD/train/dpo/llava_trainer_2stages.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb35e3ac741bb2c9"}},{"code_sha256_prefix":"735025744c1ab0cf","entry":"maybe_zero_3","repo":"richard-peng-xia/MMed-RAG","repo_kind":"official","path":"train/dpo/llava_trainer_2stages.py","file_url":"https://github.com/richard-peng-xia/MMed-RAG/blob/HEAD/train/dpo/llava_trainer_2stages.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"735025744c1ab0cf"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}