{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/improving-in-context-learning-with-small","title":"Improving In-Context Learning with Small Language Model Ensembles","arxiv_id":"2410.21868","date":"2024-10-29","proceeding":null,"authors":["M. Mehdi Mojarradi","Lingyi Yang","Robert McCraith","Adam Mahdi"],"abstract":"Large language models (LLMs) have shown impressive capabilities across various tasks, but their performance on domain-specific tasks remains limited. While methods like retrieval augmented generation and fine-tuning can help to address this, they require significant resources. In-context learning (ICL) is a cheap and efficient alternative but cannot match the accuracies of advanced methods. We present Ensemble SuperICL, a novel approach that enhances ICL by leveraging the expertise of multiple fine-tuned small language models (SLMs). Ensemble SuperICL achieves state of the art (SoTA) results on several natural language understanding benchmarks. Additionally, we test it on a medical-domain labelling task and showcase its practicality by using off-the-shelf SLMs fine-tuned on a general language task, achieving superior accuracy in large-scale data labelling compared to all baselines. Finally, we conduct an ablation study and sensitivity analyses to elucidate the underlying mechanism of Ensemble SuperICL. Our research contributes to the growing demand for efficient domain specialisation methods in LLMs, offering a cheap and effective method for practitioners.","url_abs":"https://arxiv.org/abs/2410.21868v2","url_pdf":"https://arxiv.org/pdf/2410.21868v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"improving-in-context-learning-with-small","repo_url":"https://github.com/mehdimojarradi/Ensemble-SuperICL","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"domain-labelling","task_name":"Domain Labelling"},{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"natural-language-understanding","task_name":"Natural Language Understanding"},{"task_slug":"retrieval-augmented-generation","task_name":"Retrieval-augmented Generation"},{"task_slug":"small-language-model","task_name":"Small Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.21868","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.21868"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mehdimojarradi/Ensemble-SuperICL","reach":null}],"summary":{"ran_draft_wrong":3,"unverified":1},"by_repo_kind":{"official":{"samples":4,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"64c9202dac19f4ec","entry":"extract_super_icl_prompt","repo":"mehdimojarradi/Ensemble-SuperICL","repo_kind":"official","path":"helpers.py","file_url":"https://github.com/mehdimojarradi/Ensemble-SuperICL/blob/HEAD/helpers.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"64c9202dac19f4ec"}},{"code_sha256_prefix":"941d27d960b1db7b","entry":"generate_instruction","repo":"mehdimojarradi/ensemble-supericl","repo_kind":"official","path":"main_benchmarks.py","file_url":"https://github.com/mehdimojarradi/ensemble-supericl/blob/HEAD/main_benchmarks.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"941d27d960b1db7b"}},{"code_sha256_prefix":"ec6cd0b5338651c6","entry":"get_input_template2","repo":"mehdimojarradi/ensemble-supericl","repo_kind":"official","path":"main_medmcqa.py","file_url":"https://github.com/mehdimojarradi/ensemble-supericl/blob/HEAD/main_medmcqa.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ec6cd0b5338651c6"}},{"code_sha256_prefix":"e162ab7deb54ad07","entry":"generate_instruction","repo":"mehdimojarradi/ensemble-supericl","repo_kind":"official","path":"main_medmcqa.py","file_url":"https://github.com/mehdimojarradi/ensemble-supericl/blob/HEAD/main_medmcqa.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e162ab7deb54ad07"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}