{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/commonit-commonality-aware-instruction-tuning","title":"CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions","arxiv_id":"2410.03077","date":"2024-10-04","proceeding":null,"authors":["Jun Rao","Xuebo Liu","Lian Lian","Shengjun Cheng","Yunjie Liao","Min Zhang"],"abstract":"With instruction tuning, Large Language Models (LLMs) can enhance their ability to adhere to commands. Diverging from most works focusing on data mixing, our study concentrates on enhancing the model's capabilities from the perspective of data sampling during training. Drawing inspiration from the human learning process, where it is generally easier to master solutions to similar topics through focused practice on a single type of topic, we introduce a novel instruction tuning strategy termed CommonIT: Commonality-aware Instruction Tuning. Specifically, we cluster instruction datasets into distinct groups with three proposed metrics (Task, Embedding and Length). We ensure each training mini-batch, or \"partition\", consists solely of data from a single group, which brings about both data randomness across mini-batches and intra-batch data similarity. Rigorous testing on LLaMa models demonstrates CommonIT's effectiveness in enhancing the instruction-following capabilities of LLMs through IT datasets (FLAN, CoT, and Alpaca) and models (LLaMa2-7B, Qwen2-7B, LLaMa 13B, and BLOOM 7B). CommonIT consistently boosts an average improvement of 2.1\\% on the general domain (i.e., the average score of Knowledge, Reasoning, Multilinguality and Coding) with the Length metric, and 5.2\\% on the special domain (i.e., GSM, Openfunctions and Code) with the Task metric, and 3.8\\% on the specific tasks (i.e., MMLU) with the Embedding metric. Code is available at \\url{https://github.com/raojay7/CommonIT}.","url_abs":"https://arxiv.org/abs/2410.03077v1","url_pdf":"https://arxiv.org/pdf/2410.03077v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"commonit-commonality-aware-instruction-tuning","repo_url":"https://github.com/raojay7/commonit","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"mmlu","task_name":"MMLU"}],"methods":[{"method_slug":"bloom","method_name":"BLOOM"},{"method_slug":"llama","method_name":"LLaMA"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.03077","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.03077"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/raojay7/CommonIT","reach":null}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"official":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":2,"samples":[{"code_sha256_prefix":"9998417b2747da1b","entry":"DistributedSubsetRandomSampler","repo":"raojay7/CommonIT","repo_kind":"official","path":"train_random.py","file_url":"https://github.com/raojay7/CommonIT/blob/HEAD/train_random.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9998417b2747da1b"}},{"code_sha256_prefix":"d3400db5a9eda389","entry":"MultiSubsetBatchSampler","repo":"raojay7/CommonIT","repo_kind":"official","path":"train_random.py","file_url":"https://github.com/raojay7/CommonIT/blob/HEAD/train_random.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3400db5a9eda389"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}