{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/loss-to-loss-prediction-scaling-laws-for-all","title":"Loss-to-Loss Prediction: Scaling Laws for All Datasets","arxiv_id":"2411.12925","date":"2024-11-19","proceeding":null,"authors":["David Brandfonbrener","Nikhil Anand","Nikhil Vyas","Eran Malach","Sham Kakade"],"abstract":"While scaling laws provide a reliable methodology for predicting train loss across compute scales for a single data distribution, less is known about how these predictions should change as we change the distribution. In this paper, we derive a strategy for predicting one loss from another and apply it to predict across different pre-training datasets and from pre-training data to downstream task data. Our predictions extrapolate well even at 20x the largest FLOP budget used to fit the curves. More precisely, we find that there are simple shifted power law relationships between (1) the train losses of two models trained on two separate datasets when the models are paired by training compute (train-to-train), (2) the train loss and the test loss on any downstream distribution for a single model (train-to-test), and (3) the test losses of two models trained on two separate train datasets (test-to-test). The results hold up for pre-training datasets that differ substantially (some are entirely code and others have no code at all) and across a variety of downstream tasks. Finally, we find that in some settings these shifted power law relationships can yield more accurate predictions than extrapolating single-dataset scaling laws.","url_abs":"https://arxiv.org/abs/2411.12925v1","url_pdf":"https://arxiv.org/pdf/2411.12925v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"loss-to-loss-prediction-scaling-laws-for-all","repo_url":"https://github.com/kempnerinstitute/loss-to-loss-olmo","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"loss-to-loss-prediction-scaling-laws-for-all","repo_url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"all","task_name":"All"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2411.12925","atlas_url":"https://app.syntology.ai/?focus=2411.12925","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2411.12925"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kempnerinstitute/loss-to-loss-olmo","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks","reach":{"status":"ok"}}],"summary":{"ran":1,"unverified":11},"by_repo_kind":{"official":{"samples":12,"ran":1,"repositories":2}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":9,"samples":[{"code_sha256_prefix":"f9abf6d74699cc1c","entry":"causal_attention_bias","repo":"kempnerinstitute/loss-to-loss-olmo","repo_kind":"official","path":"olmo/model.py","file_url":"https://github.com/kempnerinstitute/loss-to-loss-olmo/blob/HEAD/olmo/model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f9abf6d74699cc1c"}},{"code_sha256_prefix":"7c794f11c960d3b5","entry":"chinchilla_curve","repo":"KempnerInstitute/loss-to-loss-notebooks","repo_kind":"official","path":"utils_scaling.py","file_url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks/blob/HEAD/utils_scaling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7c794f11c960d3b5"}},{"code_sha256_prefix":"f916802c9c8b374e","entry":"fit","repo":"KempnerInstitute/loss-to-loss-notebooks","repo_kind":"official","path":"utils_scaling.py","file_url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks/blob/HEAD/utils_scaling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f916802c9c8b374e"}},{"code_sha256_prefix":"f751a49c75709d42","entry":"fix_optim_state_dict","repo":"kempnerinstitute/loss-to-loss-olmo","repo_kind":"official","path":"olmo/optim.py","file_url":"https://github.com/kempnerinstitute/loss-to-loss-olmo/blob/HEAD/olmo/optim.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"f751a49c75709d42"}},{"code_sha256_prefix":"e346f8db1d5b5881","entry":"generate_dataset","repo":"KempnerInstitute/loss-to-loss-notebooks","repo_kind":"official","path":"theory/theory_sweep_gpu.py","file_url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks/blob/HEAD/theory/theory_sweep_gpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e346f8db1d5b5881"}},{"code_sha256_prefix":"e62f060d13677db3","entry":"generate_target","repo":"KempnerInstitute/loss-to-loss-notebooks","repo_kind":"official","path":"theory/theory_sweep_gpu.py","file_url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks/blob/HEAD/theory/theory_sweep_gpu.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e62f060d13677db3"}},{"code_sha256_prefix":"f9b4804cb87312af","entry":"get_params","repo":"KempnerInstitute/loss-to-loss-notebooks","repo_kind":"official","path":"curve_fitting.py","file_url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks/blob/HEAD/curve_fitting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f9b4804cb87312af"}},{"code_sha256_prefix":"326b5329149b1741","entry":"get_simple_power_law","repo":"KempnerInstitute/loss-to-loss-notebooks","repo_kind":"official","path":"utils_plotting.py","file_url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks/blob/HEAD/utils_plotting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"326b5329149b1741"}},{"code_sha256_prefix":"5f5df690732bb25c","entry":"kaplan_curve","repo":"KempnerInstitute/loss-to-loss-notebooks","repo_kind":"official","path":"utils_scaling.py","file_url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks/blob/HEAD/utils_scaling.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5f5df690732bb25c"}},{"code_sha256_prefix":"64997ebee7737a54","entry":"load_score","repo":"kempnerinstitute/loss-to-loss-olmo","repo_kind":"official","path":"olmo/registry.py","file_url":"https://github.com/kempnerinstitute/loss-to-loss-olmo/blob/HEAD/olmo/registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"64997ebee7737a54"}},{"code_sha256_prefix":"1692167af263d3d8","entry":"plot_with_line","repo":"KempnerInstitute/loss-to-loss-notebooks","repo_kind":"official","path":"utils_plotting.py","file_url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks/blob/HEAD/utils_plotting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1692167af263d3d8"}},{"code_sha256_prefix":"85ebee271bb746fe","entry":"power_law_with_constant","repo":"KempnerInstitute/loss-to-loss-notebooks","repo_kind":"official","path":"utils_plotting.py","file_url":"https://github.com/KempnerInstitute/loss-to-loss-notebooks/blob/HEAD/utils_plotting.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85ebee271bb746fe"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}