{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lapa-latent-prompt-assist-model-for-medical","title":"LaPA: Latent Prompt Assist Model For Medical Visual Question Answering","arxiv_id":"2404.13039","date":"2024-04-19","proceeding":null,"authors":["Tiancheng Gu","Kaicheng Yang","Dongnan Liu","Weidong Cai"],"abstract":"Medical visual question answering (Med-VQA) aims to automate the prediction of correct answers for medical images and questions, thereby assisting physicians in reducing repetitive tasks and alleviating their workload. Existing approaches primarily focus on pre-training models using additional and comprehensive datasets, followed by fine-tuning to enhance performance in downstream tasks. However, there is also significant value in exploring existing models to extract clinically relevant information. In this paper, we propose the Latent Prompt Assist model (LaPA) for medical visual question answering. Firstly, we design a latent prompt generation module to generate the latent prompt with the constraint of the target answer. Subsequently, we propose a multi-modal fusion block with latent prompt fusion module that utilizes the latent prompt to extract clinical-relevant information from uni-modal and multi-modal features. Additionally, we introduce a prior knowledge fusion module to integrate the relationship between diseases and organs with the clinical-relevant information. Finally, we combine the final integrated information with image-language cross-modal information to predict the final answers. Experimental results on three publicly available Med-VQA datasets demonstrate that LaPA outperforms the state-of-the-art model ARL, achieving improvements of 1.83%, 0.63%, and 1.80% on VQA-RAD, SLAKE, and VQA-2019, respectively. The code is publicly available at https://github.com/GaryGuTC/LaPA_model.","url_abs":"https://arxiv.org/abs/2404.13039v1","url_pdf":"https://arxiv.org/pdf/2404.13039v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lapa-latent-prompt-assist-model-for-medical","repo_url":"https://github.com/garygutc/lapa_model","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"medical-visual-question-answering","task_name":"Medical Visual Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"focus","method_name":"Focus"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.13039","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.13039"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/garygutc/lapa_model","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_draft_wrong":1,"unverified":4},"by_repo_kind":{"official":{"samples":7,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"24bfe03a8fb0e475","entry":"adapt_position_encoding","repo":"garygutc/lapa_model","repo_kind":"official","path":"m3ae/modules/vision_encoders/clip_model.py","file_url":"https://github.com/garygutc/lapa_model/blob/HEAD/m3ae/modules/vision_encoders/clip_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"24bfe03a8fb0e475"}},{"code_sha256_prefix":"6eef58b15279e099","entry":"attention","repo":"garygutc/lapa_model","repo_kind":"official","path":"m3ae/modules/language_encoders/bert_model.py","file_url":"https://github.com/garygutc/lapa_model/blob/HEAD/m3ae/modules/language_encoders/bert_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6eef58b15279e099"}},{"code_sha256_prefix":"dabc5c8e7c1ae006","entry":"clones","repo":"garygutc/lapa_model","repo_kind":"official","path":"m3ae/modules/language_encoders/bert_model.py","file_url":"https://github.com/garygutc/lapa_model/blob/HEAD/m3ae/modules/language_encoders/bert_model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dabc5c8e7c1ae006"}},{"code_sha256_prefix":"7bdcea40fc62d192","entry":"build_model","repo":"garygutc/lapa_model","repo_kind":"official","path":"m3ae/modules/vision_encoders/clip_model.py","file_url":"https://github.com/garygutc/lapa_model/blob/HEAD/m3ae/modules/vision_encoders/clip_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7bdcea40fc62d192"}},{"code_sha256_prefix":"7620b2fb3b2800ef","entry":"get_pretrained_tokenizer","repo":"garygutc/lapa_model","repo_kind":"official","path":"m3ae/datamodules/base_datamodule.py","file_url":"https://github.com/garygutc/lapa_model/blob/HEAD/m3ae/datamodules/base_datamodule.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7620b2fb3b2800ef"}},{"code_sha256_prefix":"efbb0f3903b5f779","entry":"load_tf_weights_in_bert","repo":"garygutc/lapa_model","repo_kind":"official","path":"m3ae/modules/language_encoders/bert_model.py","file_url":"https://github.com/garygutc/lapa_model/blob/HEAD/m3ae/modules/language_encoders/bert_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"efbb0f3903b5f779"}},{"code_sha256_prefix":"8f1a44ab20956301","entry":"normalize_word","repo":"garygutc/lapa_model","repo_kind":"official","path":"prepro/glossary.py","file_url":"https://github.com/garygutc/lapa_model/blob/HEAD/prepro/glossary.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f1a44ab20956301"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}