{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/stability-and-generalization-of-bilevel","title":"Stability and Generalization of Bilevel Programming in Hyperparameter Optimization","arxiv_id":"2106.04188","date":"2021-06-08","proceeding":"NeurIPS 2021 12","authors":["Fan Bao","Guoqiang Wu","Chongxuan Li","Jun Zhu","Bo Zhang"],"abstract":"The (gradient-based) bilevel programming framework is widely used in hyperparameter optimization and has achieved excellent performance empirically. Previous theoretical work mainly focuses on its optimization properties, while leaving the analysis on generalization largely open. This paper attempts to address the issue by presenting an expectation bound w.r.t. the validation set based on uniform stability. Our results can explain some mysterious behaviours of the bilevel programming in practice, for instance, overfitting to the validation set. We also present an expectation bound for the classical cross-validation algorithm. Our results suggest that gradient-based algorithms can be better than cross-validation under certain conditions in a theoretical perspective. Furthermore, we prove that regularization terms in both the outer and inner levels can relieve the overfitting problem in gradient-based algorithms. In experiments on feature learning and data reweighting for noisy labels, we corroborate our theoretical findings.","url_abs":"https://arxiv.org/abs/2106.04188v2","url_pdf":"https://arxiv.org/pdf/2106.04188v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"stability-and-generalization-of-bilevel","repo_url":"https://github.com/baofff/stability_ho","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"hyperparameter-optimization","task_name":"Hyperparameter Optimization"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2106.04188","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2106.04188"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/baofff/stability_ho","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"04d429d5d8f7b262","entry":"available_devices","repo":"baofff/stability_ho","repo_kind":"official","path":"core/task_schedule.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/core/task_schedule.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"04d429d5d8f7b262"}},{"code_sha256_prefix":"90b11e74b0ed45af","entry":"create_mlp_params","repo":"baofff/stability_ho","repo_kind":"official","path":"core/mlp.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/core/mlp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"90b11e74b0ed45af"}},{"code_sha256_prefix":"b1f0685d97787668","entry":"detach","repo":"baofff/stability_ho","repo_kind":"official","path":"core/utils.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/core/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1f0685d97787668"}},{"code_sha256_prefix":"668fa61c02d4b46b","entry":"differential","repo":"baofff/stability_ho","repo_kind":"official","path":"func/differential.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/func/differential.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"668fa61c02d4b46b"}},{"code_sha256_prefix":"9c0fba82e399ffa6","entry":"format_devices","repo":"baofff/stability_ho","repo_kind":"official","path":"core/task_schedule.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/core/task_schedule.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9c0fba82e399ffa6"}},{"code_sha256_prefix":"ccb06e59708f4694","entry":"inner_product","repo":"baofff/stability_ho","repo_kind":"official","path":"func/functions.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/func/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ccb06e59708f4694"}},{"code_sha256_prefix":"a3a48551eabde6ec","entry":"judge_requires_grad","repo":"baofff/stability_ho","repo_kind":"official","path":"func/differential.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/func/differential.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3a48551eabde6ec"}},{"code_sha256_prefix":"8b7727bf81e8ddc4","entry":"mlp_forward","repo":"baofff/stability_ho","repo_kind":"official","path":"core/mlp.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/core/mlp.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b7727bf81e8ddc4"}},{"code_sha256_prefix":"c0dcc21c5f0bd9ca","entry":"outer_product","repo":"baofff/stability_ho","repo_kind":"official","path":"func/functions.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/func/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0dcc21c5f0bd9ca"}},{"code_sha256_prefix":"01a0fa7784b75fce","entry":"score_on_dataset","repo":"baofff/stability_ho","repo_kind":"official","path":"core/utils.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/core/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01a0fa7784b75fce"}},{"code_sha256_prefix":"25b53a05ad96eed9","entry":"sos","repo":"baofff/stability_ho","repo_kind":"official","path":"func/functions.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/func/functions.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"25b53a05ad96eed9"}},{"code_sha256_prefix":"7e66928c4fb0e4d8","entry":"wait_schedule","repo":"baofff/stability_ho","repo_kind":"official","path":"core/task_schedule.py","file_url":"https://github.com/baofff/stability_ho/blob/HEAD/core/task_schedule.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e66928c4fb0e4d8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}