{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/rethinking-data-selection-at-scale-random","title":"Rethinking Data Selection at Scale: Random Selection is Almost All You Need","arxiv_id":"2410.09335","date":"2024-10-12","proceeding":null,"authors":["Tingyu Xia","Bowen Yu","Kai Dang","An Yang","Yuan Wu","Yuan Tian","Yi Chang","Junyang Lin"],"abstract":"Supervised fine-tuning (SFT) is crucial for aligning Large Language Models (LLMs) with human instructions. The primary goal during SFT is to select a small yet representative subset of training data from the larger pool, such that fine-tuning with this subset achieves results comparable to or even exceeding those obtained using the entire dataset. However, most existing data selection techniques are designed for small-scale data pools, which fail to meet the demands of real-world SFT scenarios. In this paper, we replicated several self-scoring methods those that do not rely on external model assistance on two million scale datasets, and found that nearly all methods struggled to significantly outperform random selection when dealing with such large-scale data pools. Moreover, our comparisons suggest that, during SFT, diversity in data selection is more critical than simply focusing on high quality data. We also analyzed the limitations of several current approaches, explaining why they perform poorly on large-scale datasets and why they are unsuitable for such contexts. Finally, we found that filtering data by token length offers a stable and efficient method for improving results. This approach, particularly when training on long text data, proves highly beneficial for relatively weaker base models, such as Llama3.","url_abs":"https://arxiv.org/abs/2410.09335v2","url_pdf":"https://arxiv.org/pdf/2410.09335v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"rethinking-data-selection-at-scale-random","repo_url":"https://github.com/xiatingyu/sft-dataselection-at-scale","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"all","task_name":"All"}],"methods":[{"method_slug":"base","method_name":"BASE"},{"method_slug":"sft","method_name":"SFT"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.09335","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.09335"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xiatingyu/sft-dataselection-at-scale","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":4},"by_repo_kind":{"official":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":10,"samples":[{"code_sha256_prefix":"ea8f24a160d0f3ed","entry":"generate_and_tokenize_prompt","repo":"xiatingyu/sft-dataselection-at-scale","repo_kind":"official","path":"diverse/utils.py","file_url":"https://github.com/xiatingyu/sft-dataselection-at-scale/blob/HEAD/diverse/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ea8f24a160d0f3ed"}},{"code_sha256_prefix":"b2b37eed6f3f95fb","entry":"get_compression_ratio","repo":"xiatingyu/sft-dataselection-at-scale","repo_kind":"official","path":"zip/ZIP.py","file_url":"https://github.com/xiatingyu/sft-dataselection-at-scale/blob/HEAD/zip/ZIP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2b37eed6f3f95fb"}},{"code_sha256_prefix":"a17d368b7f8d0697","entry":"get_prompter","repo":"xiatingyu/sft-dataselection-at-scale","repo_kind":"official","path":"diverse/utils.py","file_url":"https://github.com/xiatingyu/sft-dataselection-at-scale/blob/HEAD/diverse/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a17d368b7f8d0697"}},{"code_sha256_prefix":"be0b30b9715eb548","entry":"load_original_data_pool","repo":"xiatingyu/sft-dataselection-at-scale","repo_kind":"official","path":"zip/ZIP.py","file_url":"https://github.com/xiatingyu/sft-dataselection-at-scale/blob/HEAD/zip/ZIP.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be0b30b9715eb548"}},{"code_sha256_prefix":"265b01725b425d23","entry":"load_sample","repo":"xiatingyu/sft-dataselection-at-scale","repo_kind":"official","path":"token_length/embedding.py","file_url":"https://github.com/xiatingyu/sft-dataselection-at-scale/blob/HEAD/token_length/embedding.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"265b01725b425d23"}},{"code_sha256_prefix":"e8d1298ca322021e","entry":"tokenize","repo":"xiatingyu/sft-dataselection-at-scale","repo_kind":"official","path":"diverse/utils.py","file_url":"https://github.com/xiatingyu/sft-dataselection-at-scale/blob/HEAD/diverse/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e8d1298ca322021e"}},{"code_sha256_prefix":"f11bec3a7dcaca3d","entry":"embed_texts_batched","repo":"xiatingyu/sft-dataselection-at-scale","repo_kind":"official","path":"token_length/embedding.py","file_url":"https://github.com/xiatingyu/sft-dataselection-at-scale/blob/HEAD/token_length/embedding.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f11bec3a7dcaca3d"}},{"code_sha256_prefix":"48b1bd5978f1385e","entry":"get_perplexity_and_embedding_part_text","repo":"xiatingyu/sft-dataselection-at-scale","repo_kind":"official","path":"IFD/cherry_seletion/data_analysis.py","file_url":"https://github.com/xiatingyu/sft-dataselection-at-scale/blob/HEAD/IFD/cherry_seletion/data_analysis.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48b1bd5978f1385e"}},{"code_sha256_prefix":"bafe0f8c61e7e6a9","entry":"get_perplexity_and_embedding_whole_text","repo":"xiatingyu/sft-dataselection-at-scale","repo_kind":"official","path":"IFD/cherry_seletion/data_analysis.py","file_url":"https://github.com/xiatingyu/sft-dataselection-at-scale/blob/HEAD/IFD/cherry_seletion/data_analysis.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bafe0f8c61e7e6a9"}},{"code_sha256_prefix":"86d72ce381c39d49","entry":"selec_data_from_corpus","repo":"xiatingyu/sft-dataselection-at-scale","repo_kind":"official","path":"zip/ZIP.py","file_url":"https://github.com/xiatingyu/sft-dataselection-at-scale/blob/HEAD/zip/ZIP.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"86d72ce381c39d49"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}