{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/datadecide-how-to-predict-best-pretraining","title":"DataDecide: How to Predict Best Pretraining Data with Small Experiments","arxiv_id":"2504.11393","date":"2025-04-15","proceeding":null,"authors":["Ian Magnusson","Nguyen Tai","Ben Bogin","David Heineman","Jena D. Hwang","Luca Soldaini","Akshita Bhagia","Jiacheng Liu","Dirk Groeneveld","Oyvind Tafjord","Noah A. Smith","Pang Wei Koh","Jesse Dodge"],"abstract":"Because large language models are expensive to pretrain on different datasets, using smaller-scale experiments to decide on data is crucial for reducing costs. Which benchmarks and methods of making decisions from observed performance at small scale most accurately predict the datasets that yield the best large models? To empower open exploration of this question, we release models, data, and evaluations in DataDecide -- the most extensive open suite of models over differences in data and scale. We conduct controlled pretraining experiments across 25 corpora with differing sources, deduplication, and filtering up to 100B tokens, model sizes up to 1B parameters, and 3 random seeds. We find that the ranking of models at a single, small size (e.g., 150M parameters) is a strong baseline for predicting best models at our larger target scale (1B) (~80% of com parisons correct). No scaling law methods among 8 baselines exceed the compute-decision frontier of single-scale predictions, but DataDecide can measure improvement in future scaling laws. We also identify that using continuous likelihood metrics as proxies in small experiments makes benchmarks including MMLU, ARC, HellaSwag, MBPP, and HumanEval >80% predictable at the target 1B scale with just 0.01% of the compute.","url_abs":"https://arxiv.org/abs/2504.11393v1","url_pdf":"https://arxiv.org/pdf/2504.11393v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"datadecide-how-to-predict-best-pretraining","repo_url":"https://github.com/gair-nlp/prox","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"arc","task_name":"ARC"},{"task_slug":"hellaswag","task_name":"HellaSwag"},{"task_slug":"humaneval","task_name":"HumanEval"},{"task_slug":"mmlu","task_name":"MMLU"},{"task_slug":null,"task_name":"mbpp"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2504.11393","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2504.11393"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/gair-nlp/prox","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":1,"ran_draft_wrong":1,"ran_honours":1,"unverified":11},"by_repo_kind":{"listed":{"samples":14,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"e05d85cc898c445c","entry":"adapter_filter","repo":"gair-nlp/prox","repo_kind":"listed","path":"train/lit_gpt/adapter_v2.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/train/lit_gpt/adapter_v2.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e05d85cc898c445c"}},{"code_sha256_prefix":"5728c74084e12d35","entry":"apply_rope","repo":"gair-nlp/prox","repo_kind":"listed","path":"train/lit_gpt/model.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/train/lit_gpt/model.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5728c74084e12d35"}},{"code_sha256_prefix":"a30c7463cb6330d7","entry":"build_rope_cache","repo":"gair-nlp/prox","repo_kind":"listed","path":"train/lit_gpt/model.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/train/lit_gpt/model.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a30c7463cb6330d7"}},{"code_sha256_prefix":"38ffb0df89b0d01f","entry":"adapter_filter","repo":"gair-nlp/prox","repo_kind":"listed","path":"train/lit_gpt/adapter.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/train/lit_gpt/adapter.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"38ffb0df89b0d01f"}},{"code_sha256_prefix":"34cce8569eec6761","entry":"execute_operations","repo":"gair-nlp/prox","repo_kind":"listed","path":"utils/chunk_utils.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/utils/chunk_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"34cce8569eec6761"}},{"code_sha256_prefix":"656a50d525064a31","entry":"generate_completions","repo":"gair-nlp/prox","repo_kind":"listed","path":"eval/utils.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/eval/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"656a50d525064a31"}},{"code_sha256_prefix":"92820145a65d0e2a","entry":"generate_unique_id","repo":"gair-nlp/prox","repo_kind":"listed","path":"utils/data_utils.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/utils/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"92820145a65d0e2a"}},{"code_sha256_prefix":"d134c88af0236263","entry":"get_next_word_predictions","repo":"gair-nlp/prox","repo_kind":"listed","path":"eval/utils.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/eval/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d134c88af0236263"}},{"code_sha256_prefix":"0af1ead737c58085","entry":"merge_chunks","repo":"gair-nlp/prox","repo_kind":"listed","path":"data_gen/tasks/apply_chunk_refining.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/data_gen/tasks/apply_chunk_refining.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0af1ead737c58085"}},{"code_sha256_prefix":"957af681b14548dd","entry":"merge_intervals","repo":"gair-nlp/prox","repo_kind":"listed","path":"utils/chunk_utils.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/utils/chunk_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"957af681b14548dd"}},{"code_sha256_prefix":"b93732570dd11835","entry":"openwebmath_adapter","repo":"gair-nlp/prox","repo_kind":"listed","path":"utils/data_utils.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/utils/data_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b93732570dd11835"}},{"code_sha256_prefix":"d029fa02758da921","entry":"score_completions","repo":"gair-nlp/prox","repo_kind":"listed","path":"eval/utils.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/eval/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d029fa02758da921"}},{"code_sha256_prefix":"60cd571547da18ee","entry":"split_doc","repo":"gair-nlp/prox","repo_kind":"listed","path":"utils/chunk_utils.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/utils/chunk_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"60cd571547da18ee"}},{"code_sha256_prefix":"774c35877b2d6534","entry":"trunc_text","repo":"gair-nlp/prox","repo_kind":"listed","path":"data_gen/tasks/apply_chunk_refining.py","file_url":"https://github.com/gair-nlp/prox/blob/HEAD/data_gen/tasks/apply_chunk_refining.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"774c35877b2d6534"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}