{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-generalization-by-controlling-label","title":"Improving Generalization by Controlling Label-Noise Information in Neural Network Weights","arxiv_id":"2002.07933","date":"2020-02-19","proceeding":"ICML 2020 1","authors":["Hrayr Harutyunyan","Kyle Reing","Greg Ver Steeg","Aram Galstyan"],"abstract":"In the presence of noisy or incorrect labels, neural networks have the undesirable tendency to memorize information about the noise. Standard regularization techniques such as dropout, weight decay or data augmentation sometimes help, but do not prevent this behavior. If one considers neural network weights as random variables that depend on the data and stochasticity of training, the amount of memorized information can be quantified with the Shannon mutual information between weights and the vector of all training labels given inputs, $I(w ; \\mathbf{y} \\mid \\mathbf{x})$. We show that for any training algorithm, low values of this term correspond to reduction in memorization of label-noise and better generalization bounds. To obtain these low values, we propose training algorithms that employ an auxiliary network that predicts gradients in the final layers of a classifier without accessing labels. We illustrate the effectiveness of our approach on versions of MNIST, CIFAR-10, and CIFAR-100 corrupted with various noise models, and on a large-scale dataset Clothing1M that has noisy labels.","url_abs":"https://arxiv.org/abs/2002.07933v2","url_pdf":"https://arxiv.org/pdf/2002.07933v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-generalization-by-controlling-label","repo_url":"https://github.com/hrayrhar/limit-label-memorization","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"generalization-bounds","task_name":"Generalization Bounds"},{"task_slug":"learning-with-noisy-labels","task_name":"Learning with noisy labels"},{"task_slug":"memorization","task_name":"Memorization"}],"methods":[{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2002.07933","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2002.07933"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hrayrhar/limit-label-memorization","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":7},"by_repo_kind":{"official":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4ae47e7c73b92c57","entry":"compute_accuracy_with_bootstrapping","repo":"hrayrhar/limit-label-memorization","repo_kind":"official","path":"modules/evaluation.py","file_url":"https://github.com/hrayrhar/limit-label-memorization/blob/HEAD/modules/evaluation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4ae47e7c73b92c57"}},{"code_sha256_prefix":"01eef8537d9a77ab","entry":"dmi","repo":"hrayrhar/limit-label-memorization","repo_kind":"official","path":"modules/losses.py","file_url":"https://github.com/hrayrhar/limit-label-memorization/blob/HEAD/modules/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01eef8537d9a77ab"}},{"code_sha256_prefix":"a78e5788e438f4b2","entry":"fw","repo":"hrayrhar/limit-label-memorization","repo_kind":"official","path":"modules/losses.py","file_url":"https://github.com/hrayrhar/limit-label-memorization/blob/HEAD/modules/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a78e5788e438f4b2"}},{"code_sha256_prefix":"f2cfffff6c7c3e88","entry":"gce","repo":"hrayrhar/limit-label-memorization","repo_kind":"official","path":"modules/losses.py","file_url":"https://github.com/hrayrhar/limit-label-memorization/blob/HEAD/modules/losses.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2cfffff6c7c3e88"}},{"code_sha256_prefix":"ba2589ac23b2a996","entry":"get_grad_noise_class","repo":"hrayrhar/limit-label-memorization","repo_kind":"official","path":"modules/nn_utils.py","file_url":"https://github.com/hrayrhar/limit-label-memorization/blob/HEAD/modules/nn_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ba2589ac23b2a996"}},{"code_sha256_prefix":"c935d78981b4c242","entry":"get_grad_replacement_class","repo":"hrayrhar/limit-label-memorization","repo_kind":"official","path":"modules/nn_utils.py","file_url":"https://github.com/hrayrhar/limit-label-memorization/blob/HEAD/modules/nn_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c935d78981b4c242"}},{"code_sha256_prefix":"f2c0aa86345bdfa8","entry":"make_resnet18k","repo":"hrayrhar/limit-label-memorization","repo_kind":"official","path":"modules/resnet18_double_descent.py","file_url":"https://github.com/hrayrhar/limit-label-memorization/blob/HEAD/modules/resnet18_double_descent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2c0aa86345bdfa8"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}