{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-usefulness-of-the-fit-on-the-test-view","title":"On the Usefulness of the Fit-on-the-Test View on Evaluating Calibration of Classifiers","arxiv_id":"2203.08958","date":"2022-03-16","proceeding":null,"authors":["Markus Kängsepp","Kaspar Valk","Meelis Kull"],"abstract":"Every uncalibrated classifier has a corresponding true calibration map that calibrates its confidence. Deviations of this idealistic map from the identity map reveal miscalibration. Such calibration errors can be reduced with many post-hoc calibration methods which fit some family of calibration maps on a validation dataset. In contrast, evaluation of calibration with the expected calibration error (ECE) on the test set does not explicitly involve fitting. However, as we demonstrate, ECE can still be viewed as if fitting a family of functions on the test data. This motivates the fit-on-the-test view on evaluation: first, approximate a calibration map on the test data, and second, quantify its distance from the identity. Exploiting this view allows us to unlock missed opportunities: (1) use the plethora of post-hoc calibration methods for evaluating calibration; (2) tune the number of bins in ECE with cross-validation. Furthermore, we introduce: (3) benchmarking on pseudo-real data where the true calibration map can be estimated very precisely; and (4) novel calibration and evaluation methods using new calibration map families PL and PL3.","url_abs":"https://arxiv.org/abs/2203.08958v3","url_pdf":"https://arxiv.org/pdf/2203.08958v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-usefulness-of-the-fit-on-the-test-view","repo_url":"https://github.com/markus93/fit-on-the-test","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"benchmarking","task_name":"Benchmarking"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2203.08958","atlas_url":"https://app.syntology.ai/?focus=2203.08958","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.08958"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/markus93/fit-on-the-test","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":15,"ran_fixture":1,"ran_violates":1,"unverified":5},"by_repo_kind":{"official":{"samples":22,"ran":17,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8f150358dd1a028b","entry":"construct_calibration_fun","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/calibration_function_derivates.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/calibration_function_derivates.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f150358dd1a028b"}},{"code_sha256_prefix":"f669081dc30a1915","entry":"construct_correlation_data_row","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/dataframe_helpers.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/dataframe_helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f669081dc30a1915"}},{"code_sha256_prefix":"0b75185643864e8d","entry":"construct_data_row","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Synthetic/run_method.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Synthetic/run_method.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0b75185643864e8d"}},{"code_sha256_prefix":"051fb62188a6ca48","entry":"construct_data_row_from_binning","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/dataframe_helpers.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/dataframe_helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"051fb62188a6ca48"}},{"code_sha256_prefix":"55ae28f90999512f","entry":"find_expected_calibration_error","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/calibration_function_derivates.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/calibration_function_derivates.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"55ae28f90999512f"}},{"code_sha256_prefix":"79d72a8e7def9af5","entry":"find_suitable_fun_amount","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/calibration_function_derivates.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/calibration_function_derivates.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"79d72a8e7def9af5"}},{"code_sha256_prefix":"0e19229d023ae722","entry":"generate_data","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/data_generation.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/data_generation.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e19229d023ae722"}},{"code_sha256_prefix":"71335b4eb90f29e0","entry":"get_cgts","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/df_unify_5m.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/df_unify_5m.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"71335b4eb90f29e0"}},{"code_sha256_prefix":"2d6d5280ab3aec05","entry":"get_data_name","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/df_unify_5m.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/df_unify_5m.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d6d5280ab3aec05"}},{"code_sha256_prefix":"e742a88077f30c11","entry":"log_encode","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/cal_methods.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/cal_methods.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e742a88077f30c11"}},{"code_sha256_prefix":"8f3819e94710b239","entry":"mirror_1d","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/kde.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/kde.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f3819e94710b239"}},{"code_sha256_prefix":"7f8c2b881b22648e","entry":"poly","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/calibration_functions.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/calibration_functions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f8c2b881b22648e"}},{"code_sha256_prefix":"115a8789d9377db2","entry":"select","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/dataframe_helpers.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/dataframe_helpers.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"115a8789d9377db2"}},{"code_sha256_prefix":"80483261c3f6087d","entry":"softmax","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/cal_methods.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/cal_methods.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"80483261c3f6087d"}},{"code_sha256_prefix":"4dc7cd36d2936a07","entry":"split_data","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/main_NN_1m_final.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/main_NN_1m_final.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4dc7cd36d2936a07"}},{"code_sha256_prefix":"645602835255b5f9","entry":"sqrt","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/calibration_functions.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/calibration_functions.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"645602835255b5f9"}},{"code_sha256_prefix":"83be324d300a4045","entry":"square","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/calibration_functions.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/calibration_functions.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"83be324d300a4045"}},{"code_sha256_prefix":"4139ecf4c8942eb3","entry":"binning_n_bins_with_crossvalidation","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/binnings.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/binnings.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4139ecf4c8942eb3"}},{"code_sha256_prefix":"bb7567171178b1d4","entry":"construct_data_row_from_binning","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Synthetic/run_method.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Synthetic/run_method.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb7567171178b1d4"}},{"code_sha256_prefix":"debb147433f25086","entry":"construct_data_row_from_raw_data","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Synthetic/run_method.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Synthetic/run_method.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"debb147433f25086"}},{"code_sha256_prefix":"8302116db328969a","entry":"get_preds_all","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/cal_methods.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/cal_methods.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8302116db328969a"}},{"code_sha256_prefix":"095a6e6431202e35","entry":"get_strs","repo":"markus93/fit-on-the-test","repo_kind":"official","path":"Experiments_Pseudo/df_unify_5m.py","file_url":"https://github.com/markus93/fit-on-the-test/blob/HEAD/Experiments_Pseudo/df_unify_5m.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"095a6e6431202e35"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}