{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/lorkd-low-rank-knowledge-decomposition-for","title":"LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models","arxiv_id":"2409.19540","date":"2024-09-29","proceeding":null,"authors":["Haolin Li","YuHang Zhou","Ziheng Zhao","Siyuan Du","Jiangchao Yao","Weidi Xie","Ya zhang","Yanfeng Wang"],"abstract":"The widespread adoption of large-scale pre-training techniques has significantly advanced the development of medical foundation models, enabling them to serve as versatile tools across a broad range of medical tasks. However, despite their strong generalization capabilities, medical foundation models pre-trained on large-scale datasets tend to suffer from domain gaps between heterogeneous data, leading to suboptimal performance on specific tasks compared to specialist models, as evidenced by previous studies. In this paper, we explore a new perspective called \"Knowledge Decomposition\" to improve the performance on specific medical tasks, which deconstructs the foundation model into multiple lightweight expert models, each dedicated to a particular anatomical region, with the aim of enhancing specialization and simultaneously reducing resource consumption. To accomplish the above objective, we propose a novel framework named Low-Rank Knowledge Decomposition (LoRKD), which explicitly separates gradients from different tasks by incorporating low-rank expert modules and efficient knowledge separation convolution. The low-rank expert modules resolve gradient conflicts between heterogeneous data from different anatomical regions, providing strong specialization at lower costs. The efficient knowledge separation convolution significantly improves algorithm efficiency by achieving knowledge separation within a single forward propagation. Extensive experimental results on segmentation and classification tasks demonstrate that our decomposed models not only achieve state-of-the-art performance but also exhibit superior transferability on downstream tasks, even surpassing the original foundation models in task-specific evaluations. The code is available at here.","url_abs":"https://arxiv.org/abs/2409.19540v1","url_pdf":"https://arxiv.org/pdf/2409.19540v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"lorkd-low-rank-knowledge-decomposition-for","repo_url":"https://github.com/mediabrain-sjtu/lorkd","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"lorkd-low-rank-knowledge-decomposition-for","repo_url":"https://github.com/tdlhl/LoRKD","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"3d-medical-imaging-segmentation","task_name":"3D Medical Imaging Segmentation"},{"task_slug":"medical-image-classification","task_name":"Medical Image Classification"}],"methods":[{"method_slug":"convolution","method_name":"Convolution"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.19540","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.19540"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mediabrain-sjtu/lorkd","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/tdlhl/LoRKD","reach":null}],"summary":{"ran":6,"ran_honours":1,"ran_fixture":2,"ran_draft_wrong":1,"unverified":3},"by_repo_kind":{"official":{"samples":13,"ran":10,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"b8c367e3d51d88b5","entry":"BaseBERT","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/query_generator.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/query_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8c367e3d51d88b5"}},{"code_sha256_prefix":"2751ff6367a1ba60","entry":"Knowledge_Encoder","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/query_generator.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/query_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2751ff6367a1ba60"}},{"code_sha256_prefix":"dc1bf240658da664","entry":"LoRALayer","repo":"mediabrain-sjtu/lorkd","repo_kind":"official","path":"peft/lora_fast_shufflenet.py","file_url":"https://github.com/mediabrain-sjtu/lorkd/blob/HEAD/peft/lora_fast_shufflenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc1bf240658da664"}},{"code_sha256_prefix":"b50742ef9e3c7903","entry":"MedCPT","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/query_generator.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/query_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b50742ef9e3c7903"}},{"code_sha256_prefix":"81774072bef023f0","entry":"MultiLoraConv2d","repo":"mediabrain-sjtu/lorkd","repo_kind":"official","path":"peft/lora_fast_shufflenet.py","file_url":"https://github.com/mediabrain-sjtu/lorkd/blob/HEAD/peft/lora_fast_shufflenet.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81774072bef023f0"}},{"code_sha256_prefix":"37f1dea898fdd023","entry":"Random_Embed","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/query_generator.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/query_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"37f1dea898fdd023"}},{"code_sha256_prefix":"62a601fb5fbbf828","entry":"compute_average_gradient","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/query_generator.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/query_generator.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"62a601fb5fbbf828"}},{"code_sha256_prefix":"168646537c6adfff","entry":"get_window_size","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/SwinUNETR.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/SwinUNETR.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"168646537c6adfff"}},{"code_sha256_prefix":"80ada9d6ef99fa0d","entry":"window_partition","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/SwinUNETR.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/SwinUNETR.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"80ada9d6ef99fa0d"}},{"code_sha256_prefix":"68a071c415424018","entry":"window_reverse","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/SwinUNETR.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/SwinUNETR.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68a071c415424018"}},{"code_sha256_prefix":"b35da59b916d1985","entry":"MyTokenizer","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/query_generator.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/query_generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b35da59b916d1985"}},{"code_sha256_prefix":"e1006ccc2b5c4dda","entry":"Query_Generator","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/query_generator.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/query_generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e1006ccc2b5c4dda"}},{"code_sha256_prefix":"3018fbd4e2cbb9cc","entry":"Text_Tower","repo":"tdlhl/LoRKD","repo_kind":"official","path":"model/query_generator.py","file_url":"https://github.com/tdlhl/LoRKD/blob/HEAD/model/query_generator.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3018fbd4e2cbb9cc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}