{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/a-consistent-and-differentiable-lp-canonical","title":"A Consistent and Differentiable Lp Canonical Calibration Error Estimator","arxiv_id":"2210.07810","date":"2022-10-13","proceeding":null,"authors":["Teodora Popordanoska","Raphael Sayer","Matthew B. Blaschko"],"abstract":"Calibrated probabilistic classifiers are models whose predicted probabilities can directly be interpreted as uncertainty estimates. It has been shown recently that deep neural networks are poorly calibrated and tend to output overconfident predictions. As a remedy, we propose a low-bias, trainable calibration error estimator based on Dirichlet kernel density estimates, which asymptotically converges to the true $L_p$ calibration error. This novel estimator enables us to tackle the strongest notion of multiclass calibration, called canonical (or distribution) calibration, while other common calibration methods are tractable only for top-label and marginal calibration. The computational complexity of our estimator is $\\mathcal{O}(n^2)$, the convergence rate is $\\mathcal{O}(n^{-1/2})$, and it is unbiased up to $\\mathcal{O}(n^{-2})$, achieved by a geometric series debiasing scheme. In practice, this means that the estimator can be applied to small subsets of data, enabling efficient estimation and mini-batch updates. The proposed method has a natural choice of kernel, and can be used to generate consistent estimates of other quantities based on conditional expectation, such as the sharpness of a probabilistic classifier. Empirical results validate the correctness of our estimator, and demonstrate its utility in canonical calibration error estimation and calibration error regularized risk minimization.","url_abs":"https://arxiv.org/abs/2210.07810v1","url_pdf":"https://arxiv.org/pdf/2210.07810v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"a-consistent-and-differentiable-lp-canonical","repo_url":"https://github.com/tpopordanoska/ece-kde","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2210.07810","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2210.07810"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/tpopordanoska/ece-kde","reach":null}],"summary":{"ran_draft_wrong":5,"ran_fixture":1,"ran":4,"unverified":6},"by_repo_kind":{"official":{"samples":16,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"9bc246bd29ace132","entry":"dirichlet_kernel","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9bc246bd29ace132"}},{"code_sha256_prefix":"b0a9bed5ef6c4741","entry":"get_ece_kde","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b0a9bed5ef6c4741"}},{"code_sha256_prefix":"33dce314122602ee","entry":"get_ece_kde","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"33dce314122602ee"}},{"code_sha256_prefix":"c78951863d26681b","entry":"get_kernel","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c78951863d26681b"}},{"code_sha256_prefix":"6b46daa5f7c4ef39","entry":"get_ratio_binary","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6b46daa5f7c4ef39"}},{"code_sha256_prefix":"0bda15c6f233eb5e","entry":"get_ratio_binary","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0bda15c6f233eb5e"}},{"code_sha256_prefix":"1279552d0b4571d1","entry":"get_ratio_canonical","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1279552d0b4571d1"}},{"code_sha256_prefix":"b6fec1cfc955ec72","entry":"get_ratio_canonical_log","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b6fec1cfc955ec72"}},{"code_sha256_prefix":"b1690d11037e0673","entry":"get_ratio_toplabel","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b1690d11037e0673"}},{"code_sha256_prefix":"75ac62b189ead8eb","entry":"isnan","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"75ac62b189ead8eb"}},{"code_sha256_prefix":"1229a437a6fcfedf","entry":"beta_kernel","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1229a437a6fcfedf"}},{"code_sha256_prefix":"5c6e3a10d18a0b2d","entry":"check_input","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c6e3a10d18a0b2d"}},{"code_sha256_prefix":"8c9800fbef285941","entry":"get_bandwidth","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8c9800fbef285941"}},{"code_sha256_prefix":"20c0edbda0d40b3a","entry":"get_kde_for_ece","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20c0edbda0d40b3a"}},{"code_sha256_prefix":"489d7eeee8368ef1","entry":"get_kde_for_ece_vect","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"489d7eeee8368ef1"}},{"code_sha256_prefix":"e3bed5deae86a699","entry":"get_ratio_marginal_vect","repo":"tpopordanoska/ece-kde","repo_kind":"official","path":"ece_kde.py","file_url":"https://github.com/tpopordanoska/ece-kde/blob/HEAD/ece_kde.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e3bed5deae86a699"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}