{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/decomposing-label-space-format-and","title":"Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning","arxiv_id":"2404.07546","date":"2024-04-11","proceeding":null,"authors":["Quanyu Long","Yin Wu","Wenya Wang","Sinno Jialin Pan"],"abstract":"In-context Learning (ICL) has emerged as a powerful capability alongside the development of scaled-up large language models (LLMs). By instructing LLMs using few-shot demonstrative examples, ICL enables them to perform a wide range of tasks without updating millions of parameters. However, the precise contributions of demonstrations towards improving end-task performance have not been thoroughly investigated in recent analytical studies. In this paper, we empirically decompose the overall performance of ICL into three dimensions, label space, format, and discrimination, and we evaluate four general-purpose LLMs across a diverse range of tasks. Counter-intuitively, we find that the demonstrations have a marginal impact on provoking discriminative knowledge of language models. However, ICL exhibits significant efficacy in regulating the label space and format, which helps LLMs respond to desired label words. We then demonstrate that this ability functions similar to detailed instructions for LLMs to follow. We additionally provide an in-depth analysis of the mechanism of retrieval helping with ICL. Our findings demonstrate that retrieving the semantically similar examples notably boosts the model's discriminative capability. However, we also observe a trade-off in selecting good in-context examples regarding label diversity.","url_abs":"https://arxiv.org/abs/2404.07546v2","url_pdf":"https://arxiv.org/pdf/2404.07546v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"decomposing-label-space-format-and","repo_url":"https://github.com/ruyue0001/decompose_icl_improvement","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.07546","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.07546"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ruyue0001/decompose_icl_improvement","reach":{"status":"ok"}}],"summary":{"ran":20,"unverified":4},"by_repo_kind":{"official":{"samples":24,"ran":20,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":24,"samples":[{"code_sha256_prefix":"aed9f558bc97e642","entry":"SARIngram","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"sari.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/sari.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"aed9f558bc97e642"}},{"code_sha256_prefix":"1eeea749ec1a866b","entry":"corpus_bleu1","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"handle_generation.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/handle_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1eeea749ec1a866b"}},{"code_sha256_prefix":"a569a12d62394751","entry":"corpus_bleu2","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"handle_generation.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/handle_generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a569a12d62394751"}},{"code_sha256_prefix":"67237d6d80179fce","entry":"create_generation_demo","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"prepare_prompt.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/prepare_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"67237d6d80179fce"}},{"code_sha256_prefix":"ec2425ceccbfcfcc","entry":"create_sample_ner","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"prepare_prompt.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/prepare_prompt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec2425ceccbfcfcc"}},{"code_sha256_prefix":"62d9ca729ee37577","entry":"get_ner","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"txt2jsonl.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/txt2jsonl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"62d9ca729ee37577"}},{"code_sha256_prefix":"116f6785089f9155","entry":"is_business","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/ag_news.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/ag_news.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"116f6785089f9155"}},{"code_sha256_prefix":"0a551c743a828127","entry":"is_entailment","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/glue_rte.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/glue_rte.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0a551c743a828127"}},{"code_sha256_prefix":"1e19118ef42994b1","entry":"is_equivalent","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/medical_question.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/medical_question.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e19118ef42994b1"}},{"code_sha256_prefix":"29e820b828a67c24","entry":"is_hate","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/hs18.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/hs18.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"29e820b828a67c24"}},{"code_sha256_prefix":"48fbfe608a52528c","entry":"is_mistral_entailment","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/glue_rte.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/glue_rte.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48fbfe608a52528c"}},{"code_sha256_prefix":"856747dbbfe12a77","entry":"is_mistral_in_space","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/hs18.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/hs18.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"856747dbbfe12a77"}},{"code_sha256_prefix":"09ad6407d9101378","entry":"is_mistral_in_space","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/medical_question.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/medical_question.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"09ad6407d9101378"}},{"code_sha256_prefix":"7b0e33bf8239b9a6","entry":"is_non_entailment","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/glue_rte.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/glue_rte.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7b0e33bf8239b9a6"}},{"code_sha256_prefix":"870e69d4013fee2e","entry":"is_non_equivalent","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/medical_question.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/medical_question.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"870e69d4013fee2e"}},{"code_sha256_prefix":"851700f22f8e29c6","entry":"is_non_hate","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/hs18.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/hs18.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"851700f22f8e29c6"}},{"code_sha256_prefix":"2ba2e8163bc35f7a","entry":"is_sci_tech","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/ag_news.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/ag_news.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2ba2e8163bc35f7a"}},{"code_sha256_prefix":"4bd4c4a432070b96","entry":"is_sports","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"post_process/ag_news.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/post_process/ag_news.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4bd4c4a432070b96"}},{"code_sha256_prefix":"db76713faf3d599e","entry":"match_original_query","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"txt2jsonl.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/txt2jsonl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"db76713faf3d599e"}},{"code_sha256_prefix":"b6a80eaa186a1dd7","entry":"match_original_query_ner","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"txt2jsonl.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/txt2jsonl.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b6a80eaa186a1dd7"}},{"code_sha256_prefix":"35effb67911bfd9a","entry":"ReadInFile","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"sari.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/sari.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35effb67911bfd9a"}},{"code_sha256_prefix":"a739e4c44244f1e2","entry":"SARIsent","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"sari.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/sari.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a739e4c44244f1e2"}},{"code_sha256_prefix":"70cb606f6be4b0c5","entry":"create_demo_ner","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"prepare_prompt.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/prepare_prompt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"70cb606f6be4b0c5"}},{"code_sha256_prefix":"c951a5a9d203c637","entry":"loader","repo":"ruyue0001/decompose_icl_improvement","repo_kind":"official","path":"generate_icl_retrieval.py","file_url":"https://github.com/ruyue0001/decompose_icl_improvement/blob/HEAD/generate_icl_retrieval.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c951a5a9d203c637"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}