{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/sample-efficient-demonstration-selection-for","title":"Sample Efficient Demonstration Selection for In-Context Learning","arxiv_id":"2506.08607","date":"2025-06-10","proceeding":null,"authors":["Kiran Purohit","V Venktesh","Sourangshu Bhattacharya","Avishek Anand"],"abstract":"The in-context learning paradigm with LLMs has been instrumental in advancing a wide range of natural language processing tasks. The selection of few-shot examples (exemplars / demonstration samples) is essential for constructing effective prompts under context-length budget constraints. In this paper, we formulate the exemplar selection task as a top-m best arms identification problem. A key challenge in this setup is the exponentially large number of arms that need to be evaluated to identify the m-best arms. We propose CASE (Challenger Arm Sampling for Exemplar selection), a novel sample-efficient selective exploration strategy that maintains a shortlist of \"challenger\" arms, which are current candidates for the top-m arms. In each iteration, only one of the arms from this shortlist or the current topm set is pulled, thereby reducing sample complexity and, consequently, the number of LLM evaluations. Furthermore, we model the scores of exemplar subsets (arms) using a parameterized linear scoring function, leading to stochastic linear bandits setting. CASE achieves remarkable efficiency gains of up to 7x speedup in runtime while requiring 7x fewer LLM calls (87% reduction) without sacrificing performance compared to state-of-the-art exemplar selection methods. We release our code and data at https://github.com/kiranpurohit/CASE","url_abs":"https://arxiv.org/abs/2506.08607v1","url_pdf":"https://arxiv.org/pdf/2506.08607v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"sample-efficient-demonstration-selection-for","repo_url":"https://github.com/kiranpurohit/case","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":null}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"}],"methods":[{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.08607","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.08607"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kiranpurohit/case","reach":null}],"summary":{"ran_draft_wrong":3,"ran":2,"unverified":3},"by_repo_kind":{"official":{"samples":8,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"f13f3af49bce9e05","entry":"clean_ans","repo":"kiranpurohit/case","repo_kind":"official","path":"Code/LLM_experiments/CASE_Gsm8K_selection.py","file_url":"https://github.com/kiranpurohit/case/blob/HEAD/Code/LLM_experiments/CASE_Gsm8K_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f13f3af49bce9e05"}},{"code_sha256_prefix":"f4cfbae1ebeb837e","entry":"get_prompt","repo":"kiranpurohit/case","repo_kind":"official","path":"Code/LLM_experiments/CASE_Gsm8K_selection.py","file_url":"https://github.com/kiranpurohit/case/blob/HEAD/Code/LLM_experiments/CASE_Gsm8K_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4cfbae1ebeb837e"}},{"code_sha256_prefix":"eef468548fe005c1","entry":"prompt_for_manual_prediction","repo":"kiranpurohit/case","repo_kind":"official","path":"Code/LLM_experiments/CASE_Gsm8K_selection.py","file_url":"https://github.com/kiranpurohit/case/blob/HEAD/Code/LLM_experiments/CASE_Gsm8K_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eef468548fe005c1"}},{"code_sha256_prefix":"4a389d1ce1f37d1b","entry":"self_con","repo":"kiranpurohit/case","repo_kind":"official","path":"Code/LLM_experiments/CASE_tabmwp_selection.py","file_url":"https://github.com/kiranpurohit/case/blob/HEAD/Code/LLM_experiments/CASE_tabmwp_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a389d1ce1f37d1b"}},{"code_sha256_prefix":"c25b31771e40684e","entry":"top_m_arm","repo":"kiranpurohit/case","repo_kind":"official","path":"Code/LLM_experiments/CASE_Gsm8K_selection.py","file_url":"https://github.com/kiranpurohit/case/blob/HEAD/Code/LLM_experiments/CASE_Gsm8K_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c25b31771e40684e"}},{"code_sha256_prefix":"175ae74476e58e2f","entry":"LLM_error_indicator","repo":"kiranpurohit/case","repo_kind":"official","path":"Code/LLM_experiments/CASE_Gsm8K_selection.py","file_url":"https://github.com/kiranpurohit/case/blob/HEAD/Code/LLM_experiments/CASE_Gsm8K_selection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"175ae74476e58e2f"}},{"code_sha256_prefix":"f2f4fbbf9e2b7fe5","entry":"get_completion","repo":"kiranpurohit/case","repo_kind":"official","path":"Code/LLM_experiments/CASE_Gsm8K_selection.py","file_url":"https://github.com/kiranpurohit/case/blob/HEAD/Code/LLM_experiments/CASE_Gsm8K_selection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f2f4fbbf9e2b7fe5"}},{"code_sha256_prefix":"8f0482096f8f0ccc","entry":"llm_output","repo":"kiranpurohit/case","repo_kind":"official","path":"Code/LLM_experiments/CASE_Gsm8K_selection.py","file_url":"https://github.com/kiranpurohit/case/blob/HEAD/Code/LLM_experiments/CASE_Gsm8K_selection.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8f0482096f8f0ccc"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}