{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/biomedical-visual-instruction-tuning-with","title":"Biomedical Visual Instruction Tuning with Clinician Preference Alignment","arxiv_id":"2406.13173","date":"2024-06-19","proceeding":null,"authors":["Hejie Cui","Lingjun Mao","Xin Liang","Jieyu Zhang","Hui Ren","Quanzheng Li","Xiang Li","Carl Yang"],"abstract":"Recent advancements in multimodal foundation models have showcased impressive capabilities in understanding and reasoning with visual and textual information. Adapting these foundation models trained for general usage to specialized domains like biomedicine requires large-scale domain-specific instruction datasets. While existing works have explored curating such datasets automatically, the resultant datasets are not explicitly aligned with domain expertise. In this work, we propose a data-centric framework, Biomedical Visual Instruction Tuning with Clinician Preference Alignment (BioMed-VITAL), that incorporates clinician preferences into both stages of generating and selecting instruction data for tuning biomedical multimodal foundation models. First, during the generation stage, we prompt the GPT-4V generator with a diverse set of clinician-selected demonstrations for preference-aligned data candidate generation. Then, during the selection phase, we train a separate selection model, which explicitly distills clinician and policy-guided model preferences into a rating function to select high-quality data for medical instruction tuning. Results show that the model tuned with the instruction-following data from our method demonstrates a significant improvement in open visual chat (18.5% relatively) and medical VQA (win rate up to 81.73%). Our instruction-following data and models are available at BioMed-VITAL.github.io.","url_abs":"https://arxiv.org/abs/2406.13173v3","url_pdf":"https://arxiv.org/pdf/2406.13173v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"biomedical-visual-instruction-tuning-with","repo_url":"https://github.com/mao1207/BioMed-VITAL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"visual-question-answering","task_name":"Visual Question Answering (VQA)"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.13173","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.13173"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mao1207/BioMed-VITAL","reach":{"status":"ok"}}],"summary":{"ran":7,"ran_draft_wrong":2,"unverified":4},"by_repo_kind":{"official":{"samples":13,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"7ce631affee8e32d","entry":"calculate_bleu","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"eval/evaluate_metrics.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/eval/evaluate_metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7ce631affee8e32d"}},{"code_sha256_prefix":"fcb74c8a86642376","entry":"compare_messages_gen","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"backbone/eval/eval_multimodal_chat_gpt_score.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/backbone/eval/eval_multimodal_chat_gpt_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fcb74c8a86642376"}},{"code_sha256_prefix":"c7eacf8297119313","entry":"conv_to_str","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"backbone/eval/eval_multimodal_chat_gpt_score.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/backbone/eval/eval_multimodal_chat_gpt_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c7eacf8297119313"}},{"code_sha256_prefix":"908505ed68ff4871","entry":"expand2square","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"backbone/mm_utils.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/backbone/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"908505ed68ff4871"}},{"code_sha256_prefix":"42a46570620cd9fa","entry":"get_chunk","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"eval/model_vqa_med.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/eval/model_vqa_med.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"42a46570620cd9fa"}},{"code_sha256_prefix":"c3ee9d07c900dd55","entry":"load_image_from_base64","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"backbone/mm_utils.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/backbone/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3ee9d07c900dd55"}},{"code_sha256_prefix":"f0f8c9c5a77c9b5b","entry":"process_images","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"backbone/mm_utils.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/backbone/mm_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f0f8c9c5a77c9b5b"}},{"code_sha256_prefix":"076c252c52cbb161","entry":"split_list","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"eval/model_vqa_med.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/eval/model_vqa_med.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"076c252c52cbb161"}},{"code_sha256_prefix":"90719ea4ae0ed586","entry":"sum_list_list","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"backbone/eval/eval_multimodal_chat_gpt_score.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/backbone/eval/eval_multimodal_chat_gpt_score.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90719ea4ae0ed586"}},{"code_sha256_prefix":"027c69e6d41cb10b","entry":"bleu","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"eval/evaluate_metrics.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/eval/evaluate_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"027c69e6d41cb10b"}},{"code_sha256_prefix":"e1820d9de24522ec","entry":"calculate_exactmatch","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"eval/evaluate_metrics.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/eval/evaluate_metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1820d9de24522ec"}},{"code_sha256_prefix":"37899f22fb191b37","entry":"pretty_print_semaphore","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"backbone/utils.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/backbone/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"37899f22fb191b37"}},{"code_sha256_prefix":"f9939a84b9a65279","entry":"violates_moderation","repo":"mao1207/BioMed-VITAL","repo_kind":"official","path":"backbone/utils.py","file_url":"https://github.com/mao1207/BioMed-VITAL/blob/HEAD/backbone/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9939a84b9a65279"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}