{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/kernelized-information-bottleneck-leads-to","title":"Kernelized information bottleneck leads to biologically plausible 3-factor Hebbian learning in deep networks","arxiv_id":"2006.07123","date":"2020-06-12","proceeding":"NeurIPS 2020 12","authors":["Roman Pogodin","Peter E. Latham"],"abstract":"The state-of-the art machine learning approach to training deep neural networks, backpropagation, is implausible for real neural networks: neurons need to know their outgoing weights; training alternates between a bottom-up forward pass (computation) and a top-down backward pass (learning); and the algorithm often needs precise labels of many data points. Biologically plausible approximations to backpropagation, such as feedback alignment, solve the weight transport problem, but not the other two. Thus, fully biologically plausible learning rules have so far remained elusive. Here we present a family of learning rules that does not suffer from any of these problems. It is motivated by the information bottleneck principle (extended with kernel methods), in which networks learn to compress the input as much as possible without sacrificing prediction of the output. The resulting rules have a 3-factor Hebbian structure: they require pre- and post-synaptic firing rates and an error signal - the third factor - consisting of a global teaching signal and a layer-specific term, both available without a top-down pass. They do not require precise labels; instead, they rely on the similarity between pairs of desired outputs. Moreover, to obtain good performance on hard problems and retain biological plausibility, our rules need divisive normalization - a known feature of biological networks. Finally, simulations show that our rules perform nearly as well as backpropagation on image classification tasks.","url_abs":"https://arxiv.org/abs/2006.07123v2","url_pdf":"https://arxiv.org/pdf/2006.07123v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"kernelized-information-bottleneck-leads-to","repo_url":"https://github.com/romanpogodin/plausible-kernelized-bottleneck","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2006.07123","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.07123"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/romanpogodin/plausible-kernelized-bottleneck","reach":null}],"summary":{"ran":2,"ran_fixture":1,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"68c6a2eddcebbdf1","entry":"HSICzyLoss","repo":"romanpogodin/plausible-kernelized-bottleneck","repo_kind":"official","path":"losses.py","file_url":"https://github.com/romanpogodin/plausible-kernelized-bottleneck/blob/HEAD/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68c6a2eddcebbdf1"}},{"code_sha256_prefix":"65ed1e34a22d8274","entry":"HSICzyLoss","repo":"romanpogodin/plausible-kernelized-bottleneck","repo_kind":"official","path":"losses.py","file_url":"https://github.com/romanpogodin/plausible-kernelized-bottleneck/blob/HEAD/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"65ed1e34a22d8274"}},{"code_sha256_prefix":"5b4431524c777f75","entry":"estimate_hsic","repo":"romanpogodin/plausible-kernelized-bottleneck","repo_kind":"official","path":"losses.py","file_url":"https://github.com/romanpogodin/plausible-kernelized-bottleneck/blob/HEAD/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5b4431524c777f75"}},{"code_sha256_prefix":"2cce205bf9b0256f","entry":"estimate_hsic_zy_objective","repo":"romanpogodin/plausible-kernelized-bottleneck","repo_kind":"official","path":"losses.py","file_url":"https://github.com/romanpogodin/plausible-kernelized-bottleneck/blob/HEAD/losses.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2cce205bf9b0256f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}