{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/early-learning-regularization-prevents","title":"Early-Learning Regularization Prevents Memorization of Noisy Labels","arxiv_id":"2007.00151","date":"2020-06-30","proceeding":"NeurIPS 2020 12","authors":["Sheng Liu","Jonathan Niles-Weed","Narges Razavian","Carlos Fernandez-Granda"],"abstract":"We propose a novel framework to perform classification via deep learning in the presence of noisy annotations. When trained on noisy labels, deep neural networks have been observed to first fit the training data with clean labels during an \"early learning\" phase, before eventually memorizing the examples with false labels. We prove that early learning and memorization are fundamental phenomena in high-dimensional classification tasks, even in simple linear models, and give a theoretical explanation in this setting. Motivated by these findings, we develop a new technique for noisy classification tasks, which exploits the progress of the early learning phase. In contrast with existing approaches, which use the model output during early learning to detect the examples with clean labels, and either ignore or attempt to correct the false labels, we take a different route and instead capitalize on early learning via regularization. There are two key elements to our approach. First, we leverage semi-supervised learning techniques to produce target probabilities based on the model outputs. Second, we design a regularization term that steers the model towards these targets, implicitly preventing memorization of the false labels. The resulting framework is shown to provide robustness to noisy annotations on several standard benchmarks and real-world datasets, where it achieves results comparable to the state of the art.","url_abs":"https://arxiv.org/abs/2007.00151v2","url_pdf":"https://arxiv.org/pdf/2007.00151v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"early-learning-regularization-prevents","repo_url":"https://github.com/shengliu66/ELR","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"early-learning-regularization-prevents","repo_url":"https://github.com/shengliu66/sop","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"classification","task_name":"General Classification"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"learning-with-noisy-labels","task_name":"Learning with noisy labels"},{"task_slug":"memorization","task_name":"Memorization"}],"methods":[{"method_slug":"elr","method_name":"ELR"}],"datasets_introduced":[],"methods_introduced":[{"slug":"elr","name":"ELR","full_name":"Early Learning Regularization"}],"results":[{"leaderboard":"/sota/image-classification-on-clothing1m","task":"Image Classification","dataset":"Clothing1M","model":"ELR+","rank_in_archive_order":14,"of":51,"metrics":{"Accuracy":"74.81%"},"uses_additional_data":true},{"leaderboard":"/sota/image-classification-on-mini-webvision-1-0","task":"Image Classification","dataset":"mini WebVision 1.0","model":"ELR+ (Inception-ResNet-v2)","rank_in_archive_order":25,"of":47,"metrics":{"ImageNet Top-1 Accuracy":"70.29","ImageNet Top-5 Accuracy":"89.76","Top-1 Accuracy":"77.78","Top-5 Accuracy":"91.68"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-100n","task":"Learning with noisy labels","dataset":"CIFAR-100N","model":"ELR+","rank_in_archive_order":6,"of":24,"metrics":{"Accuracy (mean)":"66.72"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-100n","task":"Learning with noisy labels","dataset":"CIFAR-100N","model":"ELR","rank_in_archive_order":12,"of":24,"metrics":{"Accuracy (mean)":"58.94"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-10n","task":"Learning with noisy labels","dataset":"CIFAR-10N-Aggregate","model":"ELR+","rank_in_archive_order":8,"of":26,"metrics":{"Accuracy (mean)":"94.83"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-10n","task":"Learning with noisy labels","dataset":"CIFAR-10N-Aggregate","model":"ELR","rank_in_archive_order":11,"of":26,"metrics":{"Accuracy (mean)":"92.38"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-10n-1","task":"Learning with noisy labels","dataset":"CIFAR-10N-Random1","model":"ELR+","rank_in_archive_order":7,"of":24,"metrics":{"Accuracy (mean)":"94.43"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-10n-1","task":"Learning with noisy labels","dataset":"CIFAR-10N-Random1","model":"ELR","rank_in_archive_order":9,"of":24,"metrics":{"Accuracy (mean)":"91.46"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-10n-2","task":"Learning with noisy labels","dataset":"CIFAR-10N-Random2","model":"ELR+","rank_in_archive_order":5,"of":23,"metrics":{"Accuracy (mean)":"94.20"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-10n-2","task":"Learning with noisy labels","dataset":"CIFAR-10N-Random2","model":"ELR","rank_in_archive_order":6,"of":23,"metrics":{"Accuracy (mean)":"91.61"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-10n-3","task":"Learning with noisy labels","dataset":"CIFAR-10N-Random3","model":"ELR+","rank_in_archive_order":5,"of":23,"metrics":{"Accuracy (mean)":"94.34"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-10n-3","task":"Learning with noisy labels","dataset":"CIFAR-10N-Random3","model":"ELR","rank_in_archive_order":7,"of":23,"metrics":{"Accuracy (mean)":"91.41"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-10n-worst","task":"Learning with noisy labels","dataset":"CIFAR-10N-Worst","model":"ELR+","rank_in_archive_order":8,"of":25,"metrics":{"Accuracy (mean)":"91.09"},"uses_additional_data":false},{"leaderboard":"/sota/learning-with-noisy-labels-on-cifar-10n-worst","task":"Learning with noisy labels","dataset":"CIFAR-10N-Worst","model":"ELR","rank_in_archive_order":13,"of":25,"metrics":{"Accuracy (mean)":"83.58"},"uses_additional_data":false}],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2007.00151","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2007.00151"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shengliu66/sop","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/shengliu66/ELR","reach":null}],"summary":{"ran_violates":1},"by_repo_kind":{"official":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"ab1c43d5635c2cf0","entry":"cross_entropy","repo":"shengliu66/ELR","repo_kind":"official","path":"ELR/model/loss.py","file_url":"https://github.com/shengliu66/ELR/blob/HEAD/ELR/model/loss.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab1c43d5635c2cf0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}