{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/token-based-decision-criteria-are-suboptimal","title":"Token-based Decision Criteria Are Suboptimal in In-context Learning","arxiv_id":"2406.16535","date":"2024-06-24","proceeding":null,"authors":["Hakaze Cho","Yoshihiro Sakai","Mariko Kato","Kenshiro Tanaka","Akira Ishii","Naoya Inoue"],"abstract":"In-Context Learning (ICL) typically utilizes classification criteria from output probabilities of manually selected label tokens. However, we argue that such token-based classification criteria lead to suboptimal decision boundaries, despite delicate calibrations through translation and constrained rotation applied. To address this problem, we propose Hidden Calibration, which renounces token probabilities and uses the nearest centroid classifier on the LM's last hidden states. In detail, we assign the label of the nearest centroid previously estimated from a calibration set to the test sample as the predicted label. Our experiments on 6 models and 10 classification datasets indicate that Hidden Calibration consistently outperforms current token-based baselines by about 20%~50%, achieving a strong state-of-the-art in ICL. Our further analysis demonstrates that Hidden Calibration finds better classification criteria with less inter-class overlap, and LMs provide linearly separable intra-class clusters with the help of demonstrations, which supports Hidden Calibration and gives new insights into the principle of ICL.","url_abs":"https://arxiv.org/abs/2406.16535v2","url_pdf":"https://arxiv.org/pdf/2406.16535v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"token-based-decision-criteria-are-suboptimal","repo_url":"https://github.com/hc495/Hidden_Calibration","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"classification-1","task_name":"Classification"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.16535","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.16535"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hc495/Hidden_Calibration","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"b89214893f214a17","entry":"default_prompting","repo":"hc495/Hidden_Calibration","repo_kind":"official","path":"util/prompting.py","file_url":"https://github.com/hc495/Hidden_Calibration/blob/HEAD/util/prompting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b89214893f214a17"}},{"code_sha256_prefix":"0e4cc93cabb60ecd","entry":"default_prompting_without_last_colon","repo":"hc495/Hidden_Calibration","repo_kind":"official","path":"util/prompting.py","file_url":"https://github.com/hc495/Hidden_Calibration/blob/HEAD/util/prompting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0e4cc93cabb60ecd"}},{"code_sha256_prefix":"e5024ec2270e7ac2","entry":"empty_prompting","repo":"hc495/Hidden_Calibration","repo_kind":"official","path":"util/prompting.py","file_url":"https://github.com/hc495/Hidden_Calibration/blob/HEAD/util/prompting.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e5024ec2270e7ac2"}},{"code_sha256_prefix":"632b1a204af5ef87","entry":"get_label_set_from_label_index","repo":"hc495/Hidden_Calibration","repo_kind":"official","path":"util/dataset_loader.py","file_url":"https://github.com/hc495/Hidden_Calibration/blob/HEAD/util/dataset_loader.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"632b1a204af5ef87"}},{"code_sha256_prefix":"38b84caf48189e70","entry":"pdf","repo":"hc495/Hidden_Calibration","repo_kind":"official","path":"util/calibrations.py","file_url":"https://github.com/hc495/Hidden_Calibration/blob/HEAD/util/calibrations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"38b84caf48189e70"}},{"code_sha256_prefix":"63593798e1f9db95","entry":"softmax","repo":"hc495/Hidden_Calibration","repo_kind":"official","path":"util/calibrations.py","file_url":"https://github.com/hc495/Hidden_Calibration/blob/HEAD/util/calibrations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"63593798e1f9db95"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}