{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-representation-landscape-of-few-shot","title":"The representation landscape of few-shot learning and fine-tuning in large language models","arxiv_id":"2409.03662","date":"2024-09-05","proceeding":null,"authors":["Diego Doimo","Alessandro Serra","Alessio Ansuini","Alberto Cazzaniga"],"abstract":"In-context learning (ICL) and supervised fine-tuning (SFT) are two common strategies for improving the performance of modern large language models (LLMs) on specific tasks. Despite their different natures, these strategies often lead to comparable performance gains. However, little is known about whether they induce similar representations inside LLMs. We approach this problem by analyzing the probability landscape of their hidden representations in the two cases. More specifically, we compare how LLMs solve the same question-answering task, finding that ICL and SFT create very different internal structures, in both cases undergoing a sharp transition in the middle of the network. In the first half of the network, ICL shapes interpretable representations hierarchically organized according to their semantic content. In contrast, the probability landscape obtained with SFT is fuzzier and semantically mixed. In the second half of the model, the fine-tuned representations develop probability modes that better encode the identity of answers, while the landscape of ICL representations is characterized by less defined peaks. Our approach reveals the diverse computational strategies developed inside LLMs to solve the same task across different conditions, allowing us to make a step towards designing optimal methods to extract information from language models.","url_abs":"https://arxiv.org/abs/2409.03662v2","url_pdf":"https://arxiv.org/pdf/2409.03662v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"the-representation-landscape-of-few-shot","repo_url":"https://github.com/diegodoimo/geometry_icl_finetuning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"few-shot-learning","task_name":"Few-Shot Learning"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"question-answering","task_name":"Question Answering"}],"methods":[{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2409.03662","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2409.03662"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/diegodoimo/geometry_icl_finetuning","reach":{"status":"ok"}}],"summary":{"ran":11,"unverified":2},"by_repo_kind":{"official":{"samples":13,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":13,"samples":[{"code_sha256_prefix":"b16d252479c798a0","entry":"compute_accuracy","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"finetune.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b16d252479c798a0"}},{"code_sha256_prefix":"d371140435e592be","entry":"compute_distances","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"extraction/pairwise_distances.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/extraction/pairwise_distances.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d371140435e592be"}},{"code_sha256_prefix":"95c7235973fc3b16","entry":"find_grad_accumulation_steps","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"utils/helpers_finetune.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/utils/helpers_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"95c7235973fc3b16"}},{"code_sha256_prefix":"b345237559add76d","entry":"get_cpt_steps","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"finetune.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b345237559add76d"}},{"code_sha256_prefix":"07ca7bcd15827c80","entry":"get_dataloader","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"utils/dataloader_utils.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/utils/dataloader_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07ca7bcd15827c80"}},{"code_sha256_prefix":"7c5a7b161c7346f7","entry":"get_embdims","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"extraction/helpers.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/extraction/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c5a7b161c7346f7"}},{"code_sha256_prefix":"1262b68c811b4637","entry":"get_max_memory_allocated","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"utils/helpers_finetune.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/utils/helpers_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1262b68c811b4637"}},{"code_sha256_prefix":"364acf45a359d615","entry":"get_max_memory_reserved","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"utils/helpers_finetune.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/utils/helpers_finetune.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"364acf45a359d615"}},{"code_sha256_prefix":"7909a9fdbd3ff24a","entry":"get_model","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"utils/model_utils.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/utils/model_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7909a9fdbd3ff24a"}},{"code_sha256_prefix":"1821e57029eddbc2","entry":"measure_performance","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"extraction/helpers.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/extraction/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1821e57029eddbc2"}},{"code_sha256_prefix":"9a349d02a69bd59f","entry":"return_label_overlap","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"analysis/utils.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/analysis/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a349d02a69bd59f"}},{"code_sha256_prefix":"4e34b8e1dddc6b87","entry":"evaluate","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"finetune.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/finetune.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4e34b8e1dddc6b87"}},{"code_sha256_prefix":"e9bf2efd0ef0239a","entry":"lambda_fn","repo":"diegodoimo/geometry_icl_finetuning","repo_kind":"official","path":"extract_repr.py","file_url":"https://github.com/diegodoimo/geometry_icl_finetuning/blob/HEAD/extract_repr.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e9bf2efd0ef0239a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}