{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/smaller-language-models-are-capable-of","title":"Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models","arxiv_id":"2402.10430","date":"2024-02-16","proceeding":null,"authors":["Dheeraj Mekala","Alex Nguyen","Jingbo Shang"],"abstract":"Instruction-tuning language models has become a crucial step in aligning them for general use. Typically, this process involves extensive training on large datasets, incurring high training costs. In this paper, we introduce a novel training data selection based on the learning percentage of the samples. We assert that current language models possess the capability to autonomously select high-quality training data, leading to comparable or improved performance compared to training on the entire dataset. Our experiments span different-sized models, revealing that this characteristic holds for models ranging from 1B (small) to 13B (large) in size. Moreover, we demonstrate an interesting finding that the data hardness transfers across model sizes, and a smaller 350M model can effectively curate high-quality training data with hard samples for a larger 13B model, resulting in an equally or superior instruction-tuned model compared to training on the complete dataset. Utilizing open-sourced OPT and Llama-2 models up to 13B in size, two publicly available instruction-tuning training datasets and evaluated by both automatic metrics & humans, our paper introduces a novel approach to training data selection, showcasing a more efficient alternative.","url_abs":"https://arxiv.org/abs/2402.10430v1","url_pdf":"https://arxiv.org/pdf/2402.10430v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"smaller-language-models-are-capable-of","repo_url":"https://github.com/dheeraj7596/small2large","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[],"methods":[{"method_slug":"opt","method_name":"OPT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.10430","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.10430"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dheeraj7596/small2large","reach":{"status":"ok"}}],"summary":{"ran_draft_wrong":1,"ran":3},"by_repo_kind":{"official":{"samples":4,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":4,"samples":[{"code_sha256_prefix":"d07d04439cd1d44f","entry":"jload","repo":"dheeraj7596/small2large","repo_kind":"official","path":"utils.py","file_url":"https://github.com/dheeraj7596/small2large/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d07d04439cd1d44f"}},{"code_sha256_prefix":"94a4af1edf7d18a4","entry":"jsonl_load","repo":"dheeraj7596/small2large","repo_kind":"official","path":"utils.py","file_url":"https://github.com/dheeraj7596/small2large/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"94a4af1edf7d18a4"}},{"code_sha256_prefix":"e4ab0fd6b0d41e9f","entry":"loss_per_sample","repo":"dheeraj7596/small2large","repo_kind":"official","path":"dump_pplxs.py","file_url":"https://github.com/dheeraj7596/small2large/blob/HEAD/dump_pplxs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e4ab0fd6b0d41e9f"}},{"code_sha256_prefix":"817d5f759487007d","entry":"preprocess","repo":"dheeraj7596/small2large","repo_kind":"official","path":"dump_pplxs.py","file_url":"https://github.com/dheeraj7596/small2large/blob/HEAD/dump_pplxs.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"817d5f759487007d"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}