{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2508-19999","title":"Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimation","arxiv_id":"2508.19999","date":"2025-08-27","proceeding":"EMNLP","authors":["Ziniu Zhang","Zhenshuo Zhang","Dongyue Li","Lu Wang","Jennifer Dy","Hongyang R. Zhang"],"abstract":"This paper introduces an algorithm to select demonstration examples for in-context learning of a query set. Given a set of $n$ examples, how can we quickly select $k$ out of $n$ to best serve as the conditioning for downstream inference? This problem has broad applications in prompt tuning and chain-of-thought reasoning. Since model weights remain fixed during in-context learning, previous work has sought to design methods based on the similarity of token embeddings. This work proposes a new approach based on gradients of the output taken in the input embedding space. Our approach estimates model outputs through a first-order approximation using the gradients. Then, we apply this estimation to multiple randomly sampled subsets. Finally, we aggregate the sampled subset outcomes to form an influence score for each demonstration, and select $k$ most relevant examples. This procedure only requires pre-computing model outputs and gradients once, resulting in a linear-time algorithm relative to model and training set sizes. Extensive experiments across various models and datasets validate the efficiency of our approach. We show that the gradient estimation procedure yields approximations of full inference with less than ${1}\\%$ error across six datasets. This allows us to scale up subset selection that would otherwise run full inference by up to ${37.7}\\times$ on models with up to $34$ billion parameters, and outperform existing selection methods based on input embeddings by ${11}\\%$ on average.","url_abs":"https://arxiv.org/abs/2508.19999","url_pdf":"https://arxiv.org/pdf/2508.19999","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2508.19999","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2508.19999"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/VirtuosoResearch/ICL-GradSel","reach":null}],"summary":{"ran":3,"ran_draft_wrong":2,"ran_fixture":2,"ran_violates":1},"by_repo_kind":{"found_in_text":{"samples":8,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"696ab31578bc6498","entry":"Config","repo":"VirtuosoResearch/ICL-GradSel","repo_kind":"found_in_text","path":"src/icl_linear_regression_group_lasso_demo_selection.py","file_url":"https://github.com/VirtuosoResearch/ICL-GradSel/blob/HEAD/src/icl_linear_regression_group_lasso_demo_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"696ab31578bc6498"}},{"code_sha256_prefix":"d0545ab82736c5b7","entry":"PairedTransformer","repo":"VirtuosoResearch/ICL-GradSel","repo_kind":"found_in_text","path":"src/icl_linear_regression_group_lasso_demo_selection.py","file_url":"https://github.com/VirtuosoResearch/ICL-GradSel/blob/HEAD/src/icl_linear_regression_group_lasso_demo_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d0545ab82736c5b7"}},{"code_sha256_prefix":"369d943083919008","entry":"PositionalEncoding","repo":"VirtuosoResearch/ICL-GradSel","repo_kind":"found_in_text","path":"src/icl_linear_regression_group_lasso_demo_selection.py","file_url":"https://github.com/VirtuosoResearch/ICL-GradSel/blob/HEAD/src/icl_linear_regression_group_lasso_demo_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"369d943083919008"}},{"code_sha256_prefix":"88fbfd2025e9cd11","entry":"distinct_slot_initialization","repo":"VirtuosoResearch/ICL-GradSel","repo_kind":"found_in_text","path":"src/icl_linear_regression_group_lasso_demo_selection.py","file_url":"https://github.com/VirtuosoResearch/ICL-GradSel/blob/HEAD/src/icl_linear_regression_group_lasso_demo_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"88fbfd2025e9cd11"}},{"code_sha256_prefix":"3b4f7906ce383a56","entry":"nonnegative_group_soft_threshold","repo":"VirtuosoResearch/ICL-GradSel","repo_kind":"found_in_text","path":"src/icl_linear_regression_group_lasso_demo_selection.py","file_url":"https://github.com/VirtuosoResearch/ICL-GradSel/blob/HEAD/src/icl_linear_regression_group_lasso_demo_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3b4f7906ce383a56"}},{"code_sha256_prefix":"82afecfec9f9c443","entry":"predict_query","repo":"VirtuosoResearch/ICL-GradSel","repo_kind":"found_in_text","path":"src/icl_linear_regression_group_lasso_demo_selection.py","file_url":"https://github.com/VirtuosoResearch/ICL-GradSel/blob/HEAD/src/icl_linear_regression_group_lasso_demo_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"82afecfec9f9c443"}},{"code_sha256_prefix":"a806866536d33ec5","entry":"sparse_selection_weights","repo":"VirtuosoResearch/ICL-GradSel","repo_kind":"found_in_text","path":"src/icl_linear_regression_group_lasso_demo_selection.py","file_url":"https://github.com/VirtuosoResearch/ICL-GradSel/blob/HEAD/src/icl_linear_regression_group_lasso_demo_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a806866536d33ec5"}},{"code_sha256_prefix":"0e5fe33a678c4669","entry":"task_compat_scores","repo":"VirtuosoResearch/ICL-GradSel","repo_kind":"found_in_text","path":"src/icl_linear_regression_group_lasso_demo_selection.py","file_url":"https://github.com/VirtuosoResearch/ICL-GradSel/blob/HEAD/src/icl_linear_regression_group_lasso_demo_selection.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"0e5fe33a678c4669"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_api"},"syntology_extracted_results":null}