{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/when-does-label-smoothing-help","title":"When Does Label Smoothing Help?","arxiv_id":"1906.02629","date":"2019-06-06","proceeding":"NeurIPS 2019 12","authors":["Rafael Müller","Simon Kornblith","Geoffrey Hinton"],"abstract":"The generalization and learning speed of a multi-class neural network can often be significantly improved by using soft targets that are a weighted average of the hard targets and the uniform distribution over labels. Smoothing the labels in this way prevents the network from becoming over-confident and label smoothing has been used in many state-of-the-art models, including image classification, language translation and speech recognition. Despite its widespread use, label smoothing is still poorly understood. Here we show empirically that in addition to improving generalization, label smoothing improves model calibration which can significantly improve beam-search. However, we also observe that if a teacher network is trained with label smoothing, knowledge distillation into a student network is much less effective. To explain these observations, we visualize how label smoothing changes the representations learned by the penultimate layer of the network. We show that label smoothing encourages the representations of training examples from the same class to group in tight clusters. This results in loss of information in the logits about resemblances between instances of different classes, which is necessary for distillation, but does not hurt generalization or calibration of the model's predictions.","url_abs":"https://arxiv.org/abs/1906.02629v3","url_pdf":"https://arxiv.org/pdf/1906.02629v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"when-does-label-smoothing-help","repo_url":"https://github.com/ElenaRyumina/EMO-AffectNetModel","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"when-does-label-smoothing-help","repo_url":"https://github.com/jessefokkinga/MechanismsOfAction","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}},{"paper_slug":"when-does-label-smoothing-help","repo_url":"https://github.com/seominseok0429/label-smoothing-visualization-pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"when-does-label-smoothing-help","repo_url":"https://github.com/ensta-u2is/torch-uncertainty","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"knowledge-distillation","task_name":"Knowledge Distillation"},{"task_slug":"speech-recognition","task_name":"Speech Recognition"},{"task_slug":"translation","task_name":"Translation"},{"task_slug":"image-classification","task_name":"image-classification"},{"task_slug":"speech-recognition-1","task_name":"speech-recognition"}],"methods":[{"method_slug":"knowledge-distillation","method_name":"Knowledge Distillation"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"speed","method_name":"SPEED"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1906.02629","atlas_url":"https://app.syntology.ai/?focus=1906.02629","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1906.02629"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/seominseok0429/label-smoothing-visualization-pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ensta-u2is/torch-uncertainty","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/jessefokkinga/MechanismsOfAction","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ElenaRyumina/EMO-AffectNetModel","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":5,"ran_draft_wrong":1,"unverified":1},"by_repo_kind":{"listed":{"samples":6,"ran":5,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"ef172d9e9018893b","entry":"df_group","repo":"ElenaRyumina/EMO-AffectNetModel","repo_kind":"listed","path":"functions/sequences.py","file_url":"https://github.com/ElenaRyumina/EMO-AffectNetModel/blob/HEAD/functions/sequences.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ef172d9e9018893b"}},{"code_sha256_prefix":"8d7ec010c29e813b","entry":"format_time","repo":null,"repo_kind":null,"path":null,"file_url":null,"link_basis":"identical_code_first_harvested_elsewhere","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":null,"inline_ok":false,"mcp_get_code":{"code_sha256":"8d7ec010c29e813b"}},{"code_sha256_prefix":"3cbdd3b190b16350","entry":"get_mean_and_std","repo":"seominseok0429/label-smoothing-visualization-pytorch","repo_kind":"listed","path":"utils.py","file_url":"https://github.com/seominseok0429/label-smoothing-visualization-pytorch/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3cbdd3b190b16350"}},{"code_sha256_prefix":"eee939c2db10a418","entry":"preprocess_input","repo":"ElenaRyumina/EMO-AffectNetModel","repo_kind":"listed","path":"functions/utils.py","file_url":"https://github.com/ElenaRyumina/EMO-AffectNetModel/blob/HEAD/functions/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eee939c2db10a418"}},{"code_sha256_prefix":"4f8026759621b1d5","entry":"resnet_conv_block","repo":"ElenaRyumina/EMO-AffectNetModel","repo_kind":"listed","path":"functions/get_models.py","file_url":"https://github.com/ElenaRyumina/EMO-AffectNetModel/blob/HEAD/functions/get_models.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4f8026759621b1d5"}},{"code_sha256_prefix":"816fcd30b3119901","entry":"sequences","repo":"ElenaRyumina/EMO-AffectNetModel","repo_kind":"listed","path":"functions/sequences.py","file_url":"https://github.com/ElenaRyumina/EMO-AffectNetModel/blob/HEAD/functions/sequences.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"816fcd30b3119901"}},{"code_sha256_prefix":"a396c058d14ded87","entry":"resnet_identity_block","repo":"ElenaRyumina/EMO-AffectNetModel","repo_kind":"listed","path":"functions/get_models.py","file_url":"https://github.com/ElenaRyumina/EMO-AffectNetModel/blob/HEAD/functions/get_models.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a396c058d14ded87"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}