{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-uniform-confidence-phenomenon-in-deep","title":"On the Limitations of Temperature Scaling for Distributions with Overlaps","arxiv_id":"2306.00740","date":"2023-06-01","proceeding":null,"authors":["Muthu Chidambaram","Rong Ge"],"abstract":"Despite the impressive generalization capabilities of deep neural networks, they have been repeatedly shown to be overconfident when they are wrong. Fixing this issue is known as model calibration, and has consequently received much attention in the form of modified training schemes and post-training calibration procedures such as temperature scaling. While temperature scaling is frequently used because of its simplicity, it is often outperformed by modified training schemes. In this work, we identify a specific bottleneck for the performance of temperature scaling. We show that for empirical risk minimizers for a general set of distributions in which the supports of classes have overlaps, the performance of temperature scaling degrades with the amount of overlap between classes, and asymptotically becomes no better than random when there are a large number of classes. On the other hand, we prove that optimizing a modified form of the empirical risk induced by the Mixup data augmentation technique can in fact lead to reasonably good calibration performance, showing that training-time calibration may be necessary in some situations. We also verify that our theoretical results reflect practice by showing that Mixup significantly outperforms empirical risk minimization (with respect to multiple calibration metrics) on image classification benchmarks with class overlaps introduced in the form of label noise.","url_abs":"https://arxiv.org/abs/2306.00740v3","url_pdf":"https://arxiv.org/pdf/2306.00740v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-uniform-confidence-phenomenon-in-deep","repo_url":"https://github.com/2014mchidamb/temp-scaling-limitations","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"data-augmentation","task_name":"Data Augmentation"},{"task_slug":"form","task_name":"Form"},{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"mixup","method_name":"Mixup"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2306.00740","atlas_url":"https://app.syntology.ai/?focus=2306.00740","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2306.00740"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/2014mchidamb/temp-scaling-limitations","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":9},"by_repo_kind":{"official":{"samples":9,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"322c1a3fcf7ec8c5","entry":"compute_dataset_mean_confidences","repo":"2014mchidamb/temp-scaling-limitations","repo_kind":"official","path":"utils/analysis_utils.py","file_url":"https://github.com/2014mchidamb/temp-scaling-limitations/blob/HEAD/utils/analysis_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"322c1a3fcf7ec8c5"}},{"code_sha256_prefix":"931a76b7b2cd4f06","entry":"compute_dataset_norm_statistics","repo":"2014mchidamb/temp-scaling-limitations","repo_kind":"official","path":"utils/analysis_utils.py","file_url":"https://github.com/2014mchidamb/temp-scaling-limitations/blob/HEAD/utils/analysis_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"931a76b7b2cd4f06"}},{"code_sha256_prefix":"6a9e877bf9827dc0","entry":"construct_transform","repo":"2014mchidamb/temp-scaling-limitations","repo_kind":"official","path":"utils/data_utils.py","file_url":"https://github.com/2014mchidamb/temp-scaling-limitations/blob/HEAD/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a9e877bf9827dc0"}},{"code_sha256_prefix":"96f296f4b67ed59a","entry":"get_grad_norm","repo":"2014mchidamb/temp-scaling-limitations","repo_kind":"official","path":"utils/training_utils.py","file_url":"https://github.com/2014mchidamb/temp-scaling-limitations/blob/HEAD/utils/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"96f296f4b67ed59a"}},{"code_sha256_prefix":"d543e95b1fd3f656","entry":"get_model_evaluations","repo":"2014mchidamb/temp-scaling-limitations","repo_kind":"official","path":"utils/training_utils.py","file_url":"https://github.com/2014mchidamb/temp-scaling-limitations/blob/HEAD/utils/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d543e95b1fd3f656"}},{"code_sha256_prefix":"0e87bcd416da8134","entry":"get_model_param_tensor","repo":"2014mchidamb/temp-scaling-limitations","repo_kind":"official","path":"utils/training_utils.py","file_url":"https://github.com/2014mchidamb/temp-scaling-limitations/blob/HEAD/utils/training_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e87bcd416da8134"}},{"code_sha256_prefix":"c17b086160842b78","entry":"load_fmnist","repo":"2014mchidamb/temp-scaling-limitations","repo_kind":"official","path":"utils/data_utils.py","file_url":"https://github.com/2014mchidamb/temp-scaling-limitations/blob/HEAD/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c17b086160842b78"}},{"code_sha256_prefix":"9a69802ad54de18d","entry":"load_mnist","repo":"2014mchidamb/temp-scaling-limitations","repo_kind":"official","path":"utils/data_utils.py","file_url":"https://github.com/2014mchidamb/temp-scaling-limitations/blob/HEAD/utils/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9a69802ad54de18d"}},{"code_sha256_prefix":"3b7f2eee34913659","entry":"point_nbhd_mean_confidence","repo":"2014mchidamb/temp-scaling-limitations","repo_kind":"official","path":"utils/analysis_utils.py","file_url":"https://github.com/2014mchidamb/temp-scaling-limitations/blob/HEAD/utils/analysis_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b7f2eee34913659"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}