{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/balanced-data-sampling-for-language-model","title":"Balanced Data Sampling for Language Model Training with Clustering","arxiv_id":"2402.14526","date":"2024-02-22","proceeding":null,"authors":["Yunfan Shao","Linyang Li","Zhaoye Fei","Hang Yan","Dahua Lin","Xipeng Qiu"],"abstract":"Data plays a fundamental role in the training of Large Language Models (LLMs). While attention has been paid to the collection and composition of datasets, determining the data sampling strategy in training remains an open question. Most LLMs are trained with a simple strategy, random sampling. However, this sampling strategy ignores the unbalanced nature of training data distribution, which can be sub-optimal. In this paper, we propose ClusterClip Sampling to balance the text distribution of training data for better model training. Specifically, ClusterClip Sampling utilizes data clustering to reflect the data distribution of the training set and balances the common samples and rare samples during training based on the cluster results. A repetition clip operation is introduced to mitigate the overfitting issue led by samples from certain clusters. Extensive experiments validate the effectiveness of ClusterClip Sampling, which outperforms random sampling and other cluster-based sampling variants under various training datasets and large language models.","url_abs":"https://arxiv.org/abs/2402.14526v2","url_pdf":"https://arxiv.org/pdf/2402.14526v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"balanced-data-sampling-for-language-model","repo_url":"https://github.com/choosewhatulike/cluster-clip","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"clustering","task_name":"Clustering"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"clip","method_name":"CLIP"},{"method_slug":"set","method_name":"SET"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.14526","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.14526"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/choosewhatulike/cluster-clip","reach":null}],"summary":{"ran_honours":1,"ran_fixture":1,"ran_draft_wrong":2,"unverified":1},"by_repo_kind":{"official":{"samples":5,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"091a632e49097468","entry":"distance_dot_product","repo":"choosewhatulike/cluster-clip","repo_kind":"official","path":"kmeans.py","file_url":"https://github.com/choosewhatulike/cluster-clip/blob/HEAD/kmeans.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"091a632e49097468"}},{"code_sha256_prefix":"9293dcb452ed9a4a","entry":"first_pooling","repo":"choosewhatulike/cluster-clip","repo_kind":"official","path":"run_gen_emb.py","file_url":"https://github.com/choosewhatulike/cluster-clip/blob/HEAD/run_gen_emb.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9293dcb452ed9a4a"}},{"code_sha256_prefix":"0fb325eb5b6a375b","entry":"load_embeddings","repo":"choosewhatulike/cluster-clip","repo_kind":"official","path":"kmeans.py","file_url":"https://github.com/choosewhatulike/cluster-clip/blob/HEAD/kmeans.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0fb325eb5b6a375b"}},{"code_sha256_prefix":"ee3e60cc57e00353","entry":"mean_pooling","repo":"choosewhatulike/cluster-clip","repo_kind":"official","path":"run_gen_emb.py","file_url":"https://github.com/choosewhatulike/cluster-clip/blob/HEAD/run_gen_emb.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ee3e60cc57e00353"}},{"code_sha256_prefix":"c8d10e3fa8c47d55","entry":"init_memory","repo":"choosewhatulike/cluster-clip","repo_kind":"official","path":"kmeans.py","file_url":"https://github.com/choosewhatulike/cluster-clip/blob/HEAD/kmeans.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c8d10e3fa8c47d55"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}