{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/scaling-laws-for-data-filtering-data-curation","title":"Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic","arxiv_id":"2404.07177","date":"2024-04-10","proceeding":null,"authors":["Sachin Goyal","Pratyush Maini","Zachary C. Lipton","aditi raghunathan","J. Zico Kolter"],"abstract":"Vision-language models (VLMs) are trained for thousands of GPU hours on carefully curated web datasets. In recent times, data curation has gained prominence with several works developing strategies to retain 'high-quality' subsets of 'raw' scraped data. For instance, the LAION public dataset retained only 10% of the total crawled data. However, these strategies are typically developed agnostic of the available compute for training. In this paper, we first demonstrate that making filtering decisions independent of training compute is often suboptimal: the limited high-quality data rapidly loses its utility when repeated, eventually requiring the inclusion of 'unseen' but 'lower-quality' data. To address this quality-quantity tradeoff ($\\texttt{QQT}$), we introduce neural scaling laws that account for the non-homogeneous nature of web data, an angle ignored in existing literature. Our scaling laws (i) characterize the $\\textit{differing}$ 'utility' of various quality subsets of web data; (ii) account for how utility diminishes for a data point at its 'nth' repetition; and (iii) formulate the mutual interaction of various data pools when combined, enabling the estimation of model performance on a combination of multiple data pools without ever jointly training on them. Our key message is that data curation $\\textit{cannot}$ be agnostic of the total compute that a model will be trained for. Our scaling laws allow us to curate the best possible pool for achieving top performance on Datacomp at various compute budgets, carving out a pareto-frontier for data curation. Code is available at https://github.com/locuslab/scaling_laws_data_filtering.","url_abs":"https://arxiv.org/abs/2404.07177v1","url_pdf":"https://arxiv.org/pdf/2404.07177v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"scaling-laws-for-data-filtering-data-curation","repo_url":"https://github.com/locuslab/scaling_laws_data_filtering","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":null,"task_name":"GPU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2404.07177","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2404.07177"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/locuslab/scaling_laws_data_filtering","reach":{"status":"ok"}}],"summary":{"ran":6,"unverified":2},"by_repo_kind":{"official":{"samples":8,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":8,"samples":[{"code_sha256_prefix":"3dc9bba3905befa9","entry":"func_effective_utility","repo":"locuslab/scaling_laws_data_filtering","repo_kind":"official","path":"objective.py","file_url":"https://github.com/locuslab/scaling_laws_data_filtering/blob/HEAD/objective.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3dc9bba3905befa9"}},{"code_sha256_prefix":"56e6daea0b27913a","entry":"func_effective_utility_aggregation","repo":"locuslab/scaling_laws_data_filtering","repo_kind":"official","path":"objective.py","file_url":"https://github.com/locuslab/scaling_laws_data_filtering/blob/HEAD/objective.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"56e6daea0b27913a"}},{"code_sha256_prefix":"35507707c90e4e9c","entry":"get_accuracy_from_jsonl","repo":"locuslab/scaling_laws_data_filtering","repo_kind":"official","path":"process_128_grid.py","file_url":"https://github.com/locuslab/scaling_laws_data_filtering/blob/HEAD/process_128_grid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35507707c90e4e9c"}},{"code_sha256_prefix":"387bb0d0b8f9f0ec","entry":"get_jsonl_files_tmars","repo":"locuslab/scaling_laws_data_filtering","repo_kind":"official","path":"process_128_grid.py","file_url":"https://github.com/locuslab/scaling_laws_data_filtering/blob/HEAD/process_128_grid.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"387bb0d0b8f9f0ec"}},{"code_sha256_prefix":"890a0c9c91e0da08","entry":"grid_search","repo":"locuslab/scaling_laws_data_filtering","repo_kind":"official","path":"grid_search.py","file_url":"https://github.com/locuslab/scaling_laws_data_filtering/blob/HEAD/grid_search.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"890a0c9c91e0da08"}},{"code_sha256_prefix":"d1520b6908d5986f","entry":"power","repo":"locuslab/scaling_laws_data_filtering","repo_kind":"official","path":"estimate_best_pool.py","file_url":"https://github.com/locuslab/scaling_laws_data_filtering/blob/HEAD/estimate_best_pool.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d1520b6908d5986f"}},{"code_sha256_prefix":"7aa687d516223ec6","entry":"get_avg","repo":"locuslab/scaling_laws_data_filtering","repo_kind":"official","path":"process_128_grid.py","file_url":"https://github.com/locuslab/scaling_laws_data_filtering/blob/HEAD/process_128_grid.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7aa687d516223ec6"}},{"code_sha256_prefix":"33122844de664fe1","entry":"grid_search_from_dict","repo":"locuslab/scaling_laws_data_filtering","repo_kind":"official","path":"grid_search.py","file_url":"https://github.com/locuslab/scaling_laws_data_filtering/blob/HEAD/grid_search.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"33122844de664fe1"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}