{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/knowledge-transfer-via-distillation-of","title":"Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons","arxiv_id":"1811.03233","date":"2018-11-08","proceeding":null,"authors":["Byeongho Heo","Minsik Lee","Sangdoo Yun","Jin Young Choi"],"abstract":"An activation boundary for a neuron refers to a separating hyperplane that\ndetermines whether the neuron is activated or deactivated. It has been long\nconsidered in neural networks that the activations of neurons, rather than\ntheir exact output values, play the most important role in forming\nclassification friendly partitions of the hidden feature space. However, as far\nas we know, this aspect of neural networks has not been considered in the\nliterature of knowledge transfer. In this paper, we propose a knowledge\ntransfer method via distillation of activation boundaries formed by hidden\nneurons. For the distillation, we propose an activation transfer loss that has\nthe minimum value when the boundaries generated by the student coincide with\nthose by the teacher. Since the activation transfer loss is not differentiable,\nwe design a piecewise differentiable loss approximating the activation transfer\nloss. By the proposed method, the student learns a separating boundary between\nactivation region and deactivation region formed by each neuron in the teacher.\nThrough the experiments in various aspects of knowledge transfer, it is\nverified that the proposed method outperforms the current state-of-the-art.","url_abs":"http://arxiv.org/abs/1811.03233v2","url_pdf":"http://arxiv.org/pdf/1811.03233v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"knowledge-transfer-via-distillation-of","repo_url":"https://github.com/bhheo/AB_distillation","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"knowledge-transfer-via-distillation-of","repo_url":"https://github.com/yoshitomo-matsubara/torchdistill","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"transfer-learning","task_name":"Transfer Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1811.03233","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1811.03233"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yoshitomo-matsubara/torchdistill","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bhheo/AB_distillation","reach":null}],"summary":{"unverified":1},"by_repo_kind":{"official":{"samples":1,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f7e6b413cbc14362","entry":"criterion_alternative_L2","repo":"bhheo/AB_distillation","repo_kind":"official","path":"cifar10_AB_distillation.py","file_url":"https://github.com/bhheo/AB_distillation/blob/HEAD/cifar10_AB_distillation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f7e6b413cbc14362"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}