{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/mates-model-aware-data-selection-for","title":"MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models","arxiv_id":"2406.06046","date":"2024-06-10","proceeding":null,"authors":["Zichun Yu","Spandan Das","Chenyan Xiong"],"abstract":"Pretraining data selection has the potential to improve language model pretraining efficiency by utilizing higher-quality data from massive web data corpora. Current data selection methods, which rely on either hand-crafted rules or larger reference models, are conducted statically and do not capture the evolving data preferences during pretraining. In this paper, we introduce model-aware data selection with data influence models (MATES), where a data influence model continuously adapts to the evolving data preferences of the pretraining model and then selects the data most effective for the current pretraining progress. Specifically, we collect oracle data influence by locally probing the pretraining model and fine-tune a small data influence model to approximate it accurately. The data influence model then predicts data influence over the whole pretraining corpus and selects the most influential data for the next pretraining stage. Experiments of pretraining 410M and 1B models on the C4 dataset demonstrate that MATES significantly outperforms random data selection on extensive downstream tasks. It doubles the gains achieved by the state-of-the-art data selection approach that leverages larger reference models and reduces the total FLOPs required to reach certain performances by half. Further analyses validate the effectiveness of the locally probed oracle data influence and the approximation with data influence models. Our code is open-sourced at https://github.com/cxcscmu/MATES.","url_abs":"https://arxiv.org/abs/2406.06046v2","url_pdf":"https://arxiv.org/pdf/2406.06046v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"mates-model-aware-data-selection-for","repo_url":"https://github.com/cxcscmu/mates","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[{"method_slug":"pythia","method_name":"Pythia"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2406.06046","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2406.06046"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"deterministic:regex_extraction","url":"https://github.com/cxcscmu/MATES","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cxcscmu/mates","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_violates":2,"ran":8,"ran_honours":1,"unverified":1},"by_repo_kind":{"official":{"samples":12,"ran":11,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"aa32c6faaa562b69","entry":"apply_rope","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/lit_gpt/model.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/lit_gpt/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa32c6faaa562b69"}},{"code_sha256_prefix":"7ee1c3ba71156110","entry":"build_mask_cache","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/lit_gpt/model.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/lit_gpt/model.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ee1c3ba71156110"}},{"code_sha256_prefix":"fa298642043e1282","entry":"code","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/lit_gpt/packed_dataset.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/lit_gpt/packed_dataset.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fa298642043e1282"}},{"code_sha256_prefix":"f5d9c92914e0ccfa","entry":"find_multiple","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/lit_gpt/utils.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/lit_gpt/utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f5d9c92914e0ccfa"}},{"code_sha256_prefix":"22407bf0c87de445","entry":"get_cosine_lr","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/pretrain/pretrain.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/pretrain/pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"22407bf0c87de445"}},{"code_sha256_prefix":"8e4d4ed2214edc9b","entry":"gptq_quantization","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/lit_gpt/utils.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/lit_gpt/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e4d4ed2214edc9b"}},{"code_sha256_prefix":"3b5b9d1620b5eb91","entry":"load_datasets","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/pretrain/pretrain.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/pretrain/pretrain.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3b5b9d1620b5eb91"}},{"code_sha256_prefix":"bd2356fd43fa1a5e","entry":"multinomial_num_samples_1","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/lit_gpt/gen.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/lit_gpt/gen.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bd2356fd43fa1a5e"}},{"code_sha256_prefix":"cfb725c11fe2d2f7","entry":"num_parameters","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/lit_gpt/utils.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/lit_gpt/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cfb725c11fe2d2f7"}},{"code_sha256_prefix":"668c17707928a3d7","entry":"sample","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/lit_gpt/gen.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/lit_gpt/gen.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"668c17707928a3d7"}},{"code_sha256_prefix":"1c40f8ad438abf7a","entry":"select","repo":"cxcscmu/mates","repo_kind":"official","path":"src/select_data/select_data.py","file_url":"https://github.com/cxcscmu/mates/blob/HEAD/src/select_data/select_data.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1c40f8ad438abf7a"}},{"code_sha256_prefix":"c0cabd60c81402a5","entry":"build_rope_cache","repo":"cxcscmu/MATES","repo_kind":"official","path":"src/lit_gpt/model.py","file_url":"https://github.com/cxcscmu/MATES/blob/HEAD/src/lit_gpt/model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c0cabd60c81402a5"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}