{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/reprogramming-distillation-for-medical","title":"Reprogramming Distillation for Medical Foundation Models","arxiv_id":"2407.06504","date":"2024-07-09","proceeding":null,"authors":["YuHang Zhou","Siyuan Du","Haolin Li","Jiangchao Yao","Ya zhang","Yanfeng Wang"],"abstract":"Medical foundation models pre-trained on large-scale datasets have demonstrated powerful versatile capabilities for various tasks. However, due to the gap between pre-training tasks (or modalities) and downstream tasks (or modalities), the real-world computation and speed constraints, it might not be straightforward to apply medical foundation models in the downstream scenarios. Previous methods, such as parameter efficient fine-tuning (PEFT) methods and knowledge distillation (KD) methods, are unable to simultaneously address the task (or modality) inconsistency and achieve personalized lightweight deployment under diverse real-world demands. To address the above issues, we propose a novel framework called Reprogramming Distillation (RD). On one hand, RD reprograms the original feature space of the foundation model so that it is more relevant to downstream scenarios, aligning tasks and modalities. On the other hand, through a co-training mechanism and a shared classifier, connections are established between the reprogrammed knowledge and the knowledge of student models, ensuring that the reprogrammed feature space can be smoothly mimic by the student model of different structures. Further, to reduce the randomness under different training conditions, we design a Centered Kernel Alignment (CKA) distillation to promote robust knowledge transfer. Empirically, we show that on extensive datasets, RD consistently achieve superior performance compared with previous PEFT and KD methods.","url_abs":"https://arxiv.org/abs/2407.06504v1","url_pdf":"https://arxiv.org/pdf/2407.06504v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"reprogramming-distillation-for-medical","repo_url":"https://github.com/MediaBrain-SJTU/RD","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"lightweight-deployment","task_name":"Lightweight Deployment"},{"task_slug":"transfer-learning","task_name":"Transfer Learning"},{"task_slug":"parameter-efficient-fine-tuning","task_name":"parameter-efficient fine-tuning"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2407.06504","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.06504"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MediaBrain-SJTU/RD","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":2,"ran":2,"ran_fixture":2,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"d9def42110729a85","entry":"conv1x1","repo":"MediaBrain-SJTU/RD","repo_kind":"official","path":"network/resnet.py","file_url":"https://github.com/MediaBrain-SJTU/RD/blob/HEAD/network/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d9def42110729a85"}},{"code_sha256_prefix":"fac5364e2f53c6db","entry":"conv3x3","repo":"MediaBrain-SJTU/RD","repo_kind":"official","path":"network/resnet.py","file_url":"https://github.com/MediaBrain-SJTU/RD/blob/HEAD/network/resnet.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fac5364e2f53c6db"}},{"code_sha256_prefix":"ea8083eb90046a26","entry":"conv_1x1_bn","repo":"MediaBrain-SJTU/RD","repo_kind":"official","path":"network/mobilenet_v2.py","file_url":"https://github.com/MediaBrain-SJTU/RD/blob/HEAD/network/mobilenet_v2.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea8083eb90046a26"}},{"code_sha256_prefix":"12fafb14e5d63623","entry":"radresnet50_fc","repo":"MediaBrain-SJTU/RD","repo_kind":"official","path":"network/radimagenet_resnet50.py","file_url":"https://github.com/MediaBrain-SJTU/RD/blob/HEAD/network/radimagenet_resnet50.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"12fafb14e5d63623"}},{"code_sha256_prefix":"105fa08885dc36cc","entry":"window_partition","repo":"MediaBrain-SJTU/RD","repo_kind":"official","path":"network/lvm.py","file_url":"https://github.com/MediaBrain-SJTU/RD/blob/HEAD/network/lvm.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"105fa08885dc36cc"}},{"code_sha256_prefix":"50f37d517e2be27f","entry":"window_unpartition","repo":"MediaBrain-SJTU/RD","repo_kind":"official","path":"network/lvm.py","file_url":"https://github.com/MediaBrain-SJTU/RD/blob/HEAD/network/lvm.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50f37d517e2be27f"}},{"code_sha256_prefix":"875b15ce4a332624","entry":"load_weight_for_vit_encoder","repo":"MediaBrain-SJTU/RD","repo_kind":"official","path":"network/lvm.py","file_url":"https://github.com/MediaBrain-SJTU/RD/blob/HEAD/network/lvm.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"875b15ce4a332624"}},{"code_sha256_prefix":"59dee4bfc5f2e928","entry":"radresnet50","repo":"MediaBrain-SJTU/RD","repo_kind":"official","path":"network/radimagenet_resnet50.py","file_url":"https://github.com/MediaBrain-SJTU/RD/blob/HEAD/network/radimagenet_resnet50.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"59dee4bfc5f2e928"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}