{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/data-juicer-2-0-cloud-scale-adaptive-data","title":"Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models","arxiv_id":"2501.14755","date":"2024-12-23","proceeding":null,"authors":["Daoyuan Chen","Yilun Huang","Xuchen Pan","Nana Jiang","Haibin Wang","Yilei Zhang","Ce Ge","Yushuo Chen","WenHao Zhang","Zhijian Ma","Jun Huang","Wei Lin","Yaliang Li","Bolin Ding","Jingren Zhou"],"abstract":"The burgeoning field of foundation models necessitates advanced data processing mechanisms capable of harnessing vast and valuable data with various types used by these models. Nevertheless, the current landscape presents unique challenges that traditional data processing frameworks struggle to handle effectively, particularly in handling the complexity of multimodal data. In response, we present Data-Juicer 2.0, a data processing system backed by 100+ data processing operators spanning text, image, video, and audio modalities, supporting more critical tasks including data analysis, synthesis, annotation, and foundation model post-training. With seamless compatibility and dedicated optimization for popular dataset hubs like Hugging Face and computing engines like Ray, it improves upon its predecessor in terms of usability, efficiency, and programmability. It features an easily accessible user interface layer that supports decoupled Python interactions, RESTful APIs, and conversational commands. It contains a new runtime layer optimized for adaptive execution and management across varying dataset scales, processing demands, and computational environments, while hiding unnecessary system details. Extensive empirical evaluations demonstrate Data-Juicer 2.0's remarkable performance and scalability, highlighting its capability to efficiently process TB-level data with 10k+ CPU cores. The system is publicly available and has been widely adopted in diverse research fields and real-world products such as Alibaba Cloud PAI. We actively maintain it and share insights from practical feedback, with the goal of facilitating research and application of next-generation foundation models.","url_abs":"https://arxiv.org/abs/2501.14755v2","url_pdf":"https://arxiv.org/pdf/2501.14755v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"data-juicer-2-0-cloud-scale-adaptive-data","repo_url":"https://github.com/modelscope/data-juicer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"data-juicer-2-0-cloud-scale-adaptive-data","repo_url":"https://github.com/alibaba/data-juicer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":null,"task_name":"CPU"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2501.14755","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2501.14755"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/modelscope/data-juicer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/alibaba/data-juicer","reach":null}],"summary":{"ran":1,"unverified":1},"by_repo_kind":{"listed":{"samples":2,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0177c425327ed5dd","entry":"PipelineDAG","repo":"alibaba/data-juicer","repo_kind":"listed","path":"data_juicer/core/executor/pipeline_dag.py","file_url":"https://github.com/alibaba/data-juicer/blob/HEAD/data_juicer/core/executor/pipeline_dag.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0177c425327ed5dd"}},{"code_sha256_prefix":"886792c5f9ecc160","entry":"DAGNodeStatus","repo":"alibaba/data-juicer","repo_kind":"listed","path":"data_juicer/core/executor/pipeline_dag.py","file_url":"https://github.com/alibaba/data-juicer/blob/HEAD/data_juicer/core/executor/pipeline_dag.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"886792c5f9ecc160"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}