{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-task-performance-and-model-calibration","title":"On Task Performance and Model Calibration with Supervised and Self-Ensembled In-Context Learning","arxiv_id":"2312.13772","date":"2023-12-21","proceeding":null,"authors":["Chengzu Li","Han Zhou","Goran Glavaš","Anna Korhonen","Ivan Vulić"],"abstract":"Following the standard supervised fine-tuning (SFT) paradigm, in-context learning (ICL) has become an efficient approach propelled by the recent advancements in large language models (LLMs), yielding promising performance across various tasks in few-shot data setups. However, both paradigms are prone to suffer from the critical problem of overconfidence (i.e., miscalibration), especially in such limited data setups. In this work, we deliver an in-depth analysis of the behavior across different choices of learning methods from the perspective of both performance and calibration, as well as their interplay. Through extensive controlled experiments, we find that simultaneous gains for both task performance and calibration are difficult to achieve, and the problem of miscalibration exists across all learning methods in low-resource scenarios. To address this challenging trade-off between performance and calibration, we then investigate the potential of self-ensembling techniques applied at different modeling stages (e.g., variations of in-context examples or variations in prompts or different ensembling strategies). We justify the feasibility of self-ensembling on SFT in addition to ICL, to make the predictions more calibrated and have comparable or even better performance. Our work sheds light on which learning paradigm to choose and how to enhance both task performance and calibration of LLMs.","url_abs":"https://arxiv.org/abs/2312.13772v2","url_pdf":"https://arxiv.org/pdf/2312.13772v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-task-performance-and-model-calibration","repo_url":"https://github.com/cambridgeltl/ensembled-sicl","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.13772","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.13772"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cambridgeltl/ensembled-sicl","reach":{"status":"ok"}}],"summary":{"ran":7,"unverified":3},"by_repo_kind":{"official":{"samples":10,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"ce00a88a439084fe","entry":"assert_equal","repo":"cambridgeltl/ensembled-sicl","repo_kind":"official","path":"ensemble_postprocess.py","file_url":"https://github.com/cambridgeltl/ensembled-sicl/blob/HEAD/ensemble_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ce00a88a439084fe"}},{"code_sha256_prefix":"bdb225300bc83baa","entry":"calculate_f1","repo":"cambridgeltl/ensembled-sicl","repo_kind":"official","path":"utils/evaluator.py","file_url":"https://github.com/cambridgeltl/ensembled-sicl/blob/HEAD/utils/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bdb225300bc83baa"}},{"code_sha256_prefix":"b815df4e13d2c327","entry":"check_results_num","repo":"cambridgeltl/ensembled-sicl","repo_kind":"official","path":"ensemble_postprocess.py","file_url":"https://github.com/cambridgeltl/ensembled-sicl/blob/HEAD/ensemble_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b815df4e13d2c327"}},{"code_sha256_prefix":"f6f91e27664a2312","entry":"create_peft_config","repo":"cambridgeltl/ensembled-sicl","repo_kind":"official","path":"utils/peft_scripts.py","file_url":"https://github.com/cambridgeltl/ensembled-sicl/blob/HEAD/utils/peft_scripts.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f6f91e27664a2312"}},{"code_sha256_prefix":"7d48d8f5188bc43a","entry":"load_result_json","repo":"cambridgeltl/ensembled-sicl","repo_kind":"official","path":"ensemble_postprocess.py","file_url":"https://github.com/cambridgeltl/ensembled-sicl/blob/HEAD/ensemble_postprocess.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7d48d8f5188bc43a"}},{"code_sha256_prefix":"73e30f54a5973543","entry":"precision","repo":"cambridgeltl/ensembled-sicl","repo_kind":"official","path":"utils/evaluator.py","file_url":"https://github.com/cambridgeltl/ensembled-sicl/blob/HEAD/utils/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"73e30f54a5973543"}},{"code_sha256_prefix":"9826727515a93869","entry":"recall","repo":"cambridgeltl/ensembled-sicl","repo_kind":"official","path":"utils/evaluator.py","file_url":"https://github.com/cambridgeltl/ensembled-sicl/blob/HEAD/utils/evaluator.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9826727515a93869"}},{"code_sha256_prefix":"adaa8be3ad578693","entry":"load_huggingface_model","repo":"cambridgeltl/ensembled-sicl","repo_kind":"official","path":"utils/load_model.py","file_url":"https://github.com/cambridgeltl/ensembled-sicl/blob/HEAD/utils/load_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"adaa8be3ad578693"}},{"code_sha256_prefix":"4ae9e05f3029c194","entry":"load_model","repo":"cambridgeltl/ensembled-sicl","repo_kind":"official","path":"utils/load_model.py","file_url":"https://github.com/cambridgeltl/ensembled-sicl/blob/HEAD/utils/load_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4ae9e05f3029c194"}},{"code_sha256_prefix":"618ad395c8046ae0","entry":"load_tokenizer","repo":"cambridgeltl/ensembled-sicl","repo_kind":"official","path":"utils/load_model.py","file_url":"https://github.com/cambridgeltl/ensembled-sicl/blob/HEAD/utils/load_model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"618ad395c8046ae0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}