{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/expert-knowledge-aware-image-difference-graph","title":"Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answering","arxiv_id":"2307.11986","date":"2023-07-22","proceeding":null,"authors":["Xinyue Hu","Lin Gu","Qiyuan An","Mengliang Zhang","Liangchen Liu","Kazuma Kobayashi","Tatsuya Harada","Ronald M. Summers","Yingying Zhu"],"abstract":"To contribute to automating the medical vision-language model, we propose a novel Chest-Xray Difference Visual Question Answering (VQA) task. Given a pair of main and reference images, this task attempts to answer several questions on both diseases and, more importantly, the differences between them. This is consistent with the radiologist's diagnosis practice that compares the current image with the reference before concluding the report. We collect a new dataset, namely MIMIC-Diff-VQA, including 700,703 QA pairs from 164,324 pairs of main and reference images. Compared to existing medical VQA datasets, our questions are tailored to the Assessment-Diagnosis-Intervention-Evaluation treatment procedure used by clinical professionals. Meanwhile, we also propose a novel expert knowledge-aware graph representation learning model to address this task. The proposed baseline model leverages expert knowledge such as anatomical structure prior, semantic, and spatial knowledge to construct a multi-relationship graph, representing the image differences between two images for the image difference VQA task. The dataset and code can be found at https://github.com/Holipori/MIMIC-Diff-VQA. We believe this work would further push forward the medical vision language model.","url_abs":"https://arxiv.org/abs/2307.11986v2","url_pdf":"https://arxiv.org/pdf/2307.11986v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"expert-knowledge-aware-image-difference-graph","repo_url":"https://github.com/holipori/mimic-diff-vqa","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"graph-representation-learning","task_name":"Graph Representation Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"medical-visual-question-answering","task_name":"Medical Visual Question Answering"},{"task_slug":"question-answering","task_name":"Question Answering"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"visual-question-answering-1","task_name":"Visual Question Answering"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2307.11986","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2307.11986"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/holipori/mimic-diff-vqa","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"071e6fe0c512e453","entry":"check_any_in","repo":"holipori/mimic-diff-vqa","repo_kind":"official","path":"code/question_gen.py","file_url":"https://github.com/holipori/mimic-diff-vqa/blob/HEAD/code/question_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"071e6fe0c512e453"}},{"code_sha256_prefix":"8468b020f54a228b","entry":"find_dicom","repo":"holipori/mimic-diff-vqa","repo_kind":"official","path":"code/get_mimic_all.py","file_url":"https://github.com/holipori/mimic-diff-vqa/blob/HEAD/code/get_mimic_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8468b020f54a228b"}},{"code_sha256_prefix":"0bfa225708470c8b","entry":"find_split","repo":"holipori/mimic-diff-vqa","repo_kind":"official","path":"code/get_mimic_all.py","file_url":"https://github.com/holipori/mimic-diff-vqa/blob/HEAD/code/get_mimic_all.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0bfa225708470c8b"}},{"code_sha256_prefix":"7b41ced256fa2951","entry":"get_label","repo":"holipori/mimic-diff-vqa","repo_kind":"official","path":"code/question_gen.py","file_url":"https://github.com/holipori/mimic-diff-vqa/blob/HEAD/code/question_gen.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b41ced256fa2951"}},{"code_sha256_prefix":"9e81a698f3539869","entry":"find_report","repo":"holipori/mimic-diff-vqa","repo_kind":"official","path":"code/question_gen.py","file_url":"https://github.com/holipori/mimic-diff-vqa/blob/HEAD/code/question_gen.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9e81a698f3539869"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}