{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/can-llms-tuning-methods-work-in-medical","title":"Can LLMs' Tuning Methods Work in Medical Multimodal Domain?","arxiv_id":"2403.06407","date":"2024-03-11","proceeding":null,"authors":["Jiawei Chen","Yue Jiang","Dingkang Yang","Mingcheng Li","Jinjie Wei","Ziyun Qian","Lihua Zhang"],"abstract":"While Large Language Models (LLMs) excel in world knowledge understanding, adapting them to specific subfields requires precise adjustments. Due to the model's vast scale, traditional global fine-tuning methods for large models can be computationally expensive and impact generalization. To address this challenge, a range of innovative Parameters-Efficient Fine-Tuning (PEFT) methods have emerged and achieved remarkable success in both LLMs and Large Vision-Language Models (LVLMs). In the medical domain, fine-tuning a medical Vision-Language Pretrained (VLP) model is essential for adapting it to specific tasks. Can the fine-tuning methods for large models be transferred to the medical field to enhance transfer learning efficiency? In this paper, we delve into the fine-tuning methods of LLMs and conduct extensive experiments to investigate the impact of fine-tuning methods for large models on the existing multimodal model in the medical domain from the training data level and the model structure level. We show the different impacts of fine-tuning methods for large models on medical VLMs and develop the most efficient ways to fine-tune medical VLP models. We hope this research can guide medical domain researchers in optimizing VLMs' training costs, fostering the broader application of VLMs in healthcare fields. The code and dataset have been released at https://github.com/TIMMY-CHAN/MILE.","url_abs":"https://arxiv.org/abs/2403.06407v2","url_pdf":"https://arxiv.org/pdf/2403.06407v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"can-llms-tuning-methods-work-in-medical","repo_url":"https://github.com/timmy-chan/mile","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"can-llms-tuning-methods-work-in-medical","repo_url":"https://github.com/timmy-chan/miss","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"world-knowledge","task_name":"World Knowledge"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2403.06407","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2403.06407"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/timmy-chan/mile","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/timmy-chan/miss","reach":{"status":"ok"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":3,"ran":2,"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1},"listed":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"aa81d03f07735ab8","entry":"compute_acc","repo":"timmy-chan/miss","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/timmy-chan/miss/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa81d03f07735ab8"}},{"code_sha256_prefix":"ad12a494674d23fb","entry":"compute_n_params","repo":"timmy-chan/miss","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/timmy-chan/miss/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad12a494674d23fb"}},{"code_sha256_prefix":"e71e0c413e37b037","entry":"deli","repo":"timmy-chan/miss","repo_kind":"listed","path":"ACC.py","file_url":"https://github.com/timmy-chan/miss/blob/HEAD/ACC.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e71e0c413e37b037"}},{"code_sha256_prefix":"c9a2daaf02cf5c5c","entry":"find_all_linear_names","repo":"timmy-chan/mile","repo_kind":"official","path":"train_MILE.py","file_url":"https://github.com/timmy-chan/mile/blob/HEAD/train_MILE.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c9a2daaf02cf5c5c"}},{"code_sha256_prefix":"c6ec173f19f5c34d","entry":"interpolate_pos_embed","repo":"timmy-chan/miss","repo_kind":"listed","path":"models/vit.py","file_url":"https://github.com/timmy-chan/miss/blob/HEAD/models/vit.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6ec173f19f5c34d"}},{"code_sha256_prefix":"aa5a91226269bf75","entry":"tile","repo":"timmy-chan/miss","repo_kind":"listed","path":"models/miss_vqa.py","file_url":"https://github.com/timmy-chan/miss/blob/HEAD/models/miss_vqa.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aa5a91226269bf75"}},{"code_sha256_prefix":"73cecca9f3575f09","entry":"concat_all_gather","repo":"timmy-chan/miss","repo_kind":"listed","path":"models/miss_pretrain.py","file_url":"https://github.com/timmy-chan/miss/blob/HEAD/models/miss_pretrain.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73cecca9f3575f09"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}