{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/which-examples-to-annotate-for-in-context","title":"Which Examples to Annotate for In-Context Learning? Towards Effective and Efficient Selection","arxiv_id":"2310.20046","date":"2023-10-30","proceeding":null,"authors":["Costas Mavromatis","Balasubramaniam Srinivasan","Zhengyuan Shen","Jiani Zhang","Huzefa Rangwala","Christos Faloutsos","George Karypis"],"abstract":"Large Language Models (LLMs) can adapt to new tasks via in-context learning (ICL). ICL is efficient as it does not require any parameter updates to the trained LLM, but only few annotated examples as input for the LLM. In this work, we investigate an active learning approach for ICL, where there is a limited budget for annotating examples. We propose a model-adaptive optimization-free algorithm, termed AdaICL, which identifies examples that the model is uncertain about, and performs semantic diversity-based example selection. Diversity-based sampling improves overall effectiveness, while uncertainty sampling improves budget efficiency and helps the LLM learn new information. Moreover, AdaICL poses its sampling strategy as a Maximum Coverage problem, that dynamically adapts based on the model's feedback and can be approximately solved via greedy algorithms. Extensive experiments on nine datasets and seven LLMs show that AdaICL improves performance by 4.4% accuracy points over SOTA (7.7% relative improvement), is up to 3x more budget-efficient than performing annotations uniformly at random, while it outperforms SOTA with 2x fewer ICL examples.","url_abs":"https://arxiv.org/abs/2310.20046v1","url_pdf":"https://arxiv.org/pdf/2310.20046v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"which-examples-to-annotate-for-in-context","repo_url":"https://github.com/amazon-science/adaptive-in-context-learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"active-learning","task_name":"Active Learning"},{"task_slug":"diversity","task_name":"Diversity"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2310.20046","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2310.20046"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/amazon-science/adaptive-in-context-learning","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":12,"unverified":3},"by_repo_kind":{"official":{"samples":15,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"a7424bb088354401","entry":"cluster","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"algorithms.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a7424bb088354401"}},{"code_sha256_prefix":"81689dc2c5f8dac8","entry":"density_max_coverage","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"algorithms.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"81689dc2c5f8dac8"}},{"code_sha256_prefix":"d51ed70aaef3b5d0","entry":"expand_to_aliases","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"utils.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d51ed70aaef3b5d0"}},{"code_sha256_prefix":"d72b27f74858347a","entry":"fast_votek_mod","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"algorithms.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/algorithms.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d72b27f74858347a"}},{"code_sha256_prefix":"82fab1f6e6bddec2","entry":"format_dataset","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"get_task.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/get_task.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"82fab1f6e6bddec2"}},{"code_sha256_prefix":"1e0c778236fc89b1","entry":"get_checkpoint_id","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"MetaICL/utils/utils.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/MetaICL/utils/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1e0c778236fc89b1"}},{"code_sha256_prefix":"91631e556572af73","entry":"get_sub_answers","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"utils.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"91631e556572af73"}},{"code_sha256_prefix":"6b6f0821fb467ba9","entry":"load_data","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"MetaICL/utils/data.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/MetaICL/utils/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6b6f0821fb467ba9"}},{"code_sha256_prefix":"ab97d7d0b637c027","entry":"prepro_sentence_pair","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"MetaICL/metaicl/data.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/MetaICL/metaicl/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab97d7d0b637c027"}},{"code_sha256_prefix":"daece0094ff99380","entry":"prepro_sentence_pair_single","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"MetaICL/metaicl/data.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/MetaICL/metaicl/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"daece0094ff99380"}},{"code_sha256_prefix":"071f4dd2f4d4b22c","entry":"process_mnli_examples","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"get_task.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/get_task.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"071f4dd2f4d4b22c"}},{"code_sha256_prefix":"b775e4262bf408f6","entry":"process_rte_examples","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"get_task.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/get_task.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b775e4262bf408f6"}},{"code_sha256_prefix":"ddf793183ffe3659","entry":"get_instance_length","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"prompt_retrieval.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/prompt_retrieval.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ddf793183ffe3659"}},{"code_sha256_prefix":"eff19094fe0ec70e","entry":"postprocess_text","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"main_adaptive_phases.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/main_adaptive_phases.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eff19094fe0ec70e"}},{"code_sha256_prefix":"cbf1d84d85c303cc","entry":"setup_fp16","repo":"amazon-science/adaptive-in-context-learning","repo_kind":"official","path":"MetaICL/metaicl/model.py","file_url":"https://github.com/amazon-science/adaptive-in-context-learning/blob/HEAD/MetaICL/metaicl/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cbf1d84d85c303cc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}