{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pac-bayes-information-bottleneck","title":"PAC-Bayes Information Bottleneck","arxiv_id":"2109.14509","date":"2021-09-29","proceeding":"ICLR 2022 4","authors":["Zifeng Wang","Shao-Lun Huang","Ercan E. Kuruoglu","Jimeng Sun","Xi Chen","Yefeng Zheng"],"abstract":"Understanding the source of the superior generalization ability of NNs remains one of the most important problems in ML research. There have been a series of theoretical works trying to derive non-vacuous bounds for NNs. Recently, the compression of information stored in weights (IIW) is proved to play a key role in NNs generalization based on the PAC-Bayes theorem. However, no solution of IIW has ever been provided, which builds a barrier for further investigation of the IIW's property and its potential in practical deep learning. In this paper, we propose an algorithm for the efficient approximation of IIW. Then, we build an IIW-based information bottleneck on the trade-off between accuracy and information complexity of NNs, namely PIB. From PIB, we can empirically identify the fitting to compressing phase transition during NNs' training and the concrete connection between the IIW compression and the generalization. Besides, we verify that IIW is able to explain NNs in broad cases, e.g., varying batch sizes, over-parameterization, and noisy labels. Moreover, we propose an MCMC-based algorithm to sample from the optimal weight posterior characterized by PIB, which fulfills the potential of IIW in enhancing NNs in practice.","url_abs":"https://arxiv.org/abs/2109.14509v4","url_pdf":"https://arxiv.org/pdf/2109.14509v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pac-bayes-information-bottleneck","repo_url":"https://github.com/ryanwangzf/pac-bayes-ib","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2109.14509","atlas_url":"https://app.syntology.ai/?focus=2109.14509","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2109.14509"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/RyanWangZf/PAC-Bayes-IB","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ryanwangzf/pac-bayes-ib","reach":{"status":"ok"}}],"summary":{"ran":1,"ran_violates":1,"ran_fixture":1,"ran_draft_wrong":2,"unverified":5},"by_repo_kind":{"official":{"samples":10,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"dc00bf1049269a9d","entry":"SGLD","repo":"ryanwangzf/pac-bayes-ib","repo_kind":"official","path":"src/pib_utils.py","file_url":"https://github.com/ryanwangzf/pac-bayes-ib/blob/HEAD/src/pib_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dc00bf1049269a9d"}},{"code_sha256_prefix":"162e5c220b43fc38","entry":"adjust_learning_rate","repo":"ryanwangzf/pac-bayes-ib","repo_kind":"official","path":"src/pib_utils.py","file_url":"https://github.com/ryanwangzf/pac-bayes-ib/blob/HEAD/src/pib_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"162e5c220b43fc38"}},{"code_sha256_prefix":"e2137b2a6d2e103c","entry":"eval_metric","repo":"ryanwangzf/pac-bayes-ib","repo_kind":"official","path":"src/pib_utils.py","file_url":"https://github.com/ryanwangzf/pac-bayes-ib/blob/HEAD/src/pib_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e2137b2a6d2e103c"}},{"code_sha256_prefix":"793985ca63e12a72","entry":"eval_metric_binary","repo":"ryanwangzf/pac-bayes-ib","repo_kind":"official","path":"src/pib_utils.py","file_url":"https://github.com/ryanwangzf/pac-bayes-ib/blob/HEAD/src/pib_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"793985ca63e12a72"}},{"code_sha256_prefix":"05619bf46a5078e5","entry":"predict","repo":"ryanwangzf/pac-bayes-ib","repo_kind":"official","path":"src/pib_utils.py","file_url":"https://github.com/ryanwangzf/pac-bayes-ib/blob/HEAD/src/pib_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"05619bf46a5078e5"}},{"code_sha256_prefix":"7e84c8fa1c587e30","entry":"load_model","repo":"ryanwangzf/pac-bayes-ib","repo_kind":"official","path":"src/pib_utils.py","file_url":"https://github.com/ryanwangzf/pac-bayes-ib/blob/HEAD/src/pib_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e84c8fa1c587e30"}},{"code_sha256_prefix":"fd653d0c452a188d","entry":"save_model","repo":"ryanwangzf/pac-bayes-ib","repo_kind":"official","path":"src/pib_utils.py","file_url":"https://github.com/ryanwangzf/pac-bayes-ib/blob/HEAD/src/pib_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fd653d0c452a188d"}},{"code_sha256_prefix":"18184c74464685b3","entry":"train","repo":"ryanwangzf/pac-bayes-ib","repo_kind":"official","path":"src/pib_utils.py","file_url":"https://github.com/ryanwangzf/pac-bayes-ib/blob/HEAD/src/pib_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"18184c74464685b3"}},{"code_sha256_prefix":"5cb1898d18432034","entry":"train_SGLD","repo":"ryanwangzf/pac-bayes-ib","repo_kind":"official","path":"src/pib_utils.py","file_url":"https://github.com/ryanwangzf/pac-bayes-ib/blob/HEAD/src/pib_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5cb1898d18432034"}},{"code_sha256_prefix":"a5f603988167977c","entry":"train_pib","repo":"ryanwangzf/pac-bayes-ib","repo_kind":"official","path":"src/pib_utils.py","file_url":"https://github.com/ryanwangzf/pac-bayes-ib/blob/HEAD/src/pib_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a5f603988167977c"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}