{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/new-horizons-in-parameter-regularization-a","title":"Improving Deep Learning Optimization through Constrained Parameter Regularization","arxiv_id":"2311.09058","date":"2023-11-15","proceeding":null,"authors":["Jörg K. H. Franke","Michael Hefenbrock","Gregor Koehler","Frank Hutter"],"abstract":"Regularization is a critical component in deep learning. The most commonly used approach, weight decay, applies a constant penalty coefficient uniformly across all parameters. This may be overly restrictive for some parameters, while insufficient for others. To address this, we present Constrained Parameter Regularization (CPR) as an alternative to traditional weight decay. Unlike the uniform application of a single penalty, CPR enforces an upper bound on a statistical measure, such as the L2-norm, of individual parameter matrices. Consequently, learning becomes a constraint optimization problem, which we tackle using an adaptation of the augmented Lagrangian method. CPR introduces only a minor runtime overhead and only requires setting an upper bound. We propose simple yet efficient mechanisms for initializing this bound, making CPR rely on no hyperparameter or one, akin to weight decay. Our empirical studies on computer vision and language modeling tasks demonstrate CPR's effectiveness. The results show that CPR can outperform traditional weight decay and increase performance in pre-training and fine-tuning.","url_abs":"https://arxiv.org/abs/2311.09058v4","url_pdf":"https://arxiv.org/pdf/2311.09058v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"new-horizons-in-parameter-regularization-a","repo_url":"https://github.com/automl/cpr","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"deep-learning","task_name":"Deep Learning"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2311.09058","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2311.09058"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/automl/cpr","reach":null}],"summary":{"ran":2,"ran_draft_wrong":2,"unverified":9},"by_repo_kind":{"official":{"samples":13,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"77e74503b4a6ddf3","entry":"_disable_dynamo_if_unsupported","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"77e74503b4a6ddf3"}},{"code_sha256_prefix":"8d0c76131beabbb9","entry":"_get_capturable_supported_devices","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8d0c76131beabbb9"}},{"code_sha256_prefix":"2f3d97790ecb08b8","entry":"group_parameters_for_cpr_optimizer","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2f3d97790ecb08b8"}},{"code_sha256_prefix":"2d82e29243858703","entry":"wd_group_named_parameters","repo":"automl/cpr","repo_kind":"official","path":"examples/train_cifar100_task.py","file_url":"https://github.com/automl/cpr/blob/HEAD/examples/train_cifar100_task.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d82e29243858703"}},{"code_sha256_prefix":"20c73d219b728426","entry":"AdamCPR","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"20c73d219b728426"}},{"code_sha256_prefix":"ee694b63b0bc4895","entry":"_multi_tensor_adamcpr","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee694b63b0bc4895"}},{"code_sha256_prefix":"3e3e242295adb363","entry":"_single_tensor_adamcpr","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e3e242295adb363"}},{"code_sha256_prefix":"a14f930855d7e136","entry":"adamcpr","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a14f930855d7e136"}},{"code_sha256_prefix":"5d028ca59f740e80","entry":"huber_update","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5d028ca59f740e80"}},{"code_sha256_prefix":"f5fe436ea316e71f","entry":"l1_update","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f5fe436ea316e71f"}},{"code_sha256_prefix":"3babf488a0693b0b","entry":"l2_update","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3babf488a0693b0b"}},{"code_sha256_prefix":"08254562e66103c1","entry":"single_initilize_kappa","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"08254562e66103c1"}},{"code_sha256_prefix":"ddf970467b201f00","entry":"std_update","repo":"automl/cpr","repo_kind":"official","path":"pytorch_cpr/adamcpr.py","file_url":"https://github.com/automl/cpr/blob/HEAD/pytorch_cpr/adamcpr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ddf970467b201f00"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}