{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/adaptive-data-optimization-dynamic-sample","title":"Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws","arxiv_id":"2410.11820","date":"2024-10-15","proceeding":null,"authors":["Yiding Jiang","Allan Zhou","Zhili Feng","Sadhika Malladi","J. Zico Kolter"],"abstract":"The composition of pretraining data is a key determinant of foundation models' performance, but there is no standard guideline for allocating a limited computational budget across different data sources. Most current approaches either rely on extensive experiments with smaller models or dynamic data adjustments that also require proxy models, both of which significantly increase the workflow complexity and computational overhead. In this paper, we introduce Adaptive Data Optimization (ADO), an algorithm that optimizes data distributions in an online fashion, concurrent with model training. Unlike existing techniques, ADO does not require external knowledge, proxy models, or modifications to the model update. Instead, ADO uses per-domain scaling laws to estimate the learning potential of each domain during training and adjusts the data mixture accordingly, making it more scalable and easier to integrate. Experiments demonstrate that ADO can achieve comparable or better performance than prior methods while maintaining computational efficiency across different computation scales, offering a practical solution for dynamically adjusting data distribution without sacrificing flexibility or increasing costs. Beyond its practical benefits, ADO also provides a new perspective on data collection strategies via scaling laws.","url_abs":"https://arxiv.org/abs/2410.11820v1","url_pdf":"https://arxiv.org/pdf/2410.11820v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"adaptive-data-optimization-dynamic-sample","repo_url":"https://github.com/yidingjiang/ado","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":null},{"paper_slug":"adaptive-data-optimization-dynamic-sample","repo_url":"https://github.com/eth-easl/mixtera","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"computational-efficiency","task_name":"Computational Efficiency"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2410.11820","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.11820"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/yidingjiang/ado","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/eth-easl/mixtera","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran_violates":1,"ran_draft_wrong":1,"unverified":5},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"listed":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"61efae1dab3abd01","entry":"clip_min_probability","repo":"yidingjiang/ado","repo_kind":"official","path":"src/data_selectors/ado.py","file_url":"https://github.com/yidingjiang/ado/blob/HEAD/src/data_selectors/ado.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"61efae1dab3abd01"}},{"code_sha256_prefix":"b2591cd2d6a28012","entry":"huber_loss","repo":"yidingjiang/ado","repo_kind":"official","path":"src/data_selectors/ado.py","file_url":"https://github.com/yidingjiang/ado/blob/HEAD/src/data_selectors/ado.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b2591cd2d6a28012"}},{"code_sha256_prefix":"a53948498ea45ede","entry":"logsumexp","repo":"yidingjiang/ado","repo_kind":"official","path":"src/data_selectors/ado.py","file_url":"https://github.com/yidingjiang/ado/blob/HEAD/src/data_selectors/ado.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a53948498ea45ede"}},{"code_sha256_prefix":"2d3d99ed392e678e","entry":"calc_func","repo":"eth-easl/mixtera","repo_kind":"listed","path":"mixtera_integrationtests/utils.py","file_url":"https://github.com/eth-easl/mixtera/blob/HEAD/mixtera_integrationtests/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2d3d99ed392e678e"}},{"code_sha256_prefix":"1d182bd6530f45e2","entry":"get_expected_js_and_html_samples","repo":"eth-easl/mixtera","repo_kind":"listed","path":"mixtera_integrationtests/utils.py","file_url":"https://github.com/eth-easl/mixtera/blob/HEAD/mixtera_integrationtests/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1d182bd6530f45e2"}},{"code_sha256_prefix":"320f58d334e9e37f","entry":"read_bytes","repo":"eth-easl/mixtera","repo_kind":"listed","path":"mixtera/network/network_utils.py","file_url":"https://github.com/eth-easl/mixtera/blob/HEAD/mixtera/network/network_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"320f58d334e9e37f"}},{"code_sha256_prefix":"f36c9c890aa5c624","entry":"read_bytes_obj","repo":"eth-easl/mixtera","repo_kind":"listed","path":"mixtera/network/network_utils.py","file_url":"https://github.com/eth-easl/mixtera/blob/HEAD/mixtera/network/network_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f36c9c890aa5c624"}},{"code_sha256_prefix":"ca2334cf3e3905b1","entry":"read_int","repo":"eth-easl/mixtera","repo_kind":"listed","path":"mixtera/network/network_utils.py","file_url":"https://github.com/eth-easl/mixtera/blob/HEAD/mixtera/network/network_utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ca2334cf3e3905b1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}