{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/computing-nonvacuous-generalization-bounds","title":"Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data","arxiv_id":"1703.11008","date":"2017-03-31","proceeding":null,"authors":["Gintare Karolina Dziugaite","Daniel M. Roy"],"abstract":"One of the defining properties of deep learning is that models are chosen to\nhave many more parameters than available training data. In light of this\ncapacity for overfitting, it is remarkable that simple algorithms like SGD\nreliably return solutions with low test error. One roadblock to explaining\nthese phenomena in terms of implicit regularization, structural properties of\nthe solution, and/or easiness of the data is that many learning bounds are\nquantitatively vacuous when applied to networks learned by SGD in this \"deep\nlearning\" regime. Logically, in order to explain generalization, we need\nnonvacuous bounds. We return to an idea by Langford and Caruana (2001), who\nused PAC-Bayes bounds to compute nonvacuous numerical bounds on generalization\nerror for stochastic two-layer two-hidden-unit neural networks via a\nsensitivity analysis. By optimizing the PAC-Bayes bound directly, we are able\nto extend their approach and obtain nonvacuous generalization bounds for deep\nstochastic neural network classifiers with millions of parameters trained on\nonly tens of thousands of examples. We connect our findings to recent and old\nwork on flat minima and MDL-based explanations of generalization.","url_abs":"http://arxiv.org/abs/1703.11008v2","url_pdf":"http://arxiv.org/pdf/1703.11008v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"computing-nonvacuous-generalization-bounds","repo_url":"https://github.com/gkdziugaite/pacbayes-opt","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"computing-nonvacuous-generalization-bounds","repo_url":"https://github.com/grasp-lyrl/sloppy","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"computing-nonvacuous-generalization-bounds","repo_url":"https://github.com/AliceDeLorenci/Nonvacuous-Generalization-Bounds-for-DNNs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"unanswered"}}],"tasks":[{"task_slug":"generalization-bounds","task_name":"Generalization Bounds"}],"methods":[{"method_slug":"sgd","method_name":"SGD"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1703.11008","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1703.11008"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/AliceDeLorenci/Nonvacuous-Generalization-Bounds-for-DNNs","reach":{"status":"unanswered"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gkdziugaite/pacbayes-opt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/grasp-lyrl/sloppy","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"listed":{"samples":3,"ran":2,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"922db1f94e3abe45","entry":"run_sgd","repo":"gkdziugaite/pacbayes-opt","repo_kind":"listed","path":"snn/experiments/run_sgd.py","file_url":"https://github.com/gkdziugaite/pacbayes-opt/blob/HEAD/snn/experiments/run_sgd.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"922db1f94e3abe45"}},{"code_sha256_prefix":"500971d824c3e2d9","entry":"sec","repo":"grasp-lyrl/sloppy","repo_kind":"listed","path":"bayes1.py","file_url":"https://github.com/grasp-lyrl/sloppy/blob/HEAD/bayes1.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"500971d824c3e2d9"}},{"code_sha256_prefix":"0112fd9162a57434","entry":"val","repo":"grasp-lyrl/sloppy","repo_kind":"listed","path":"bayes1.py","file_url":"https://github.com/grasp-lyrl/sloppy/blob/HEAD/bayes1.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0112fd9162a57434"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}