{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/feature-based-one-for-all-a-universal","title":"Feature-based One-For-All: A Universal Framework for Heterogeneous Knowledge Distillation","arxiv_id":"2501.08885","date":"2025-01-15","proceeding":null,"authors":["Jhe-Hao Lin","Yi Yao","Chan-Feng Hsu","HongXia Xie","Hong-Han Shuai","Wen-Huang Cheng"],"abstract":"Knowledge distillation (KD) involves transferring knowledge from a pre-trained heavy teacher model to a lighter student model, thereby reducing the inference cost while maintaining comparable effectiveness. Prior KD techniques typically assume homogeneity between the teacher and student models. However, as technology advances, a wide variety of architectures have emerged, ranging from initial Convolutional Neural Networks (CNNs) to Vision Transformers (ViTs), and Multi-Level Perceptrons (MLPs). Consequently, developing a universal KD framework compatible with any architecture has become an important research topic. In this paper, we introduce a feature-based one-for-all (FOFA) KD framework to enable feature distillation across diverse architecture. Our framework comprises two key components. First, we design prompt tuning blocks that incorporate student feedback, allowing teacher features to adapt to the student model's learning process. Second, we propose region-aware attention to mitigate the view mismatch problem between heterogeneous architecture. By leveraging these two modules, effective distillation of intermediate features can be achieved across heterogeneous architectures. Extensive experiments on CIFAR, ImageNet, and COCO demonstrate the superiority of the proposed method.","url_abs":"https://arxiv.org/abs/2501.08885v1","url_pdf":"https://arxiv.org/pdf/2501.08885v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[],"tasks":[{"task_slug":"all","task_name":"All"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"}],"methods":[{"method_slug":"attention","method_name":"Attention"},{"method_slug":"softmax","method_name":"Softmax"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.08885","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.08885"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/jimmylin0979/PAT","reach":null}],"summary":{"ran":8,"ran_honours":1,"ran_fixture":2,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"found_in_text":{"samples":16,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7faaedcc21f0004d","entry":"BaseDistiller","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7faaedcc21f0004d"}},{"code_sha256_prefix":"373d5549a4931491","entry":"CNN2CNNProjector","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"373d5549a4931491"}},{"code_sha256_prefix":"58ec827bec449ba6","entry":"CNN2ViTProjector","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"58ec827bec449ba6"}},{"code_sha256_prefix":"7761c53ac0e3680a","entry":"RAProjector","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7761c53ac0e3680a"}},{"code_sha256_prefix":"897882a5e8a88c61","entry":"RegionAttention","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"897882a5e8a88c61"}},{"code_sha256_prefix":"0546ce0adeb02bde","entry":"SEBlock","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0546ce0adeb02bde"}},{"code_sha256_prefix":"d352000861a8fd36","entry":"ViT2CNNProjector","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d352000861a8fd36"}},{"code_sha256_prefix":"fd196a0c189320de","entry":"ViT2ViTProjector","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"fd196a0c189320de"}},{"code_sha256_prefix":"df1050a734942c85","entry":"get_positional_encoding","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"df1050a734942c85"}},{"code_sha256_prefix":"f24a50ec4768698e","entry":"get_projector_module","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f24a50ec4768698e"}},{"code_sha256_prefix":"f493ed098c0c2ecb","entry":"hcl_loss","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f493ed098c0c2ecb"}},{"code_sha256_prefix":"23388af4327e43c1","entry":"kd_loss","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"23388af4327e43c1"}},{"code_sha256_prefix":"0466efd8ed0a4c22","entry":"register_distiller","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0466efd8ed0a4c22"}},{"code_sha256_prefix":"f4cf6aac98290b55","entry":"PAT","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f4cf6aac98290b55"}},{"code_sha256_prefix":"e4459f636cf3355d","entry":"PromptBlock","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e4459f636cf3355d"}},{"code_sha256_prefix":"93f26e689279411d","entry":"init_weights","repo":"jimmylin0979/PAT","repo_kind":"found_in_text","path":"distillers/pat.py","file_url":"https://github.com/jimmylin0979/PAT/blob/HEAD/distillers/pat.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"93f26e689279411d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}