{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/plumber-diagnosing-and-removing-performance","title":"Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines","arxiv_id":"2111.04131","date":"2021-11-07","proceeding":null,"authors":["Michael Kuchnik","Ana Klimovic","Jiri Simsa","Virginia Smith","George Amvrosiadis"],"abstract":"Input pipelines, which ingest and transform input data, are an essential part of training Machine Learning (ML) models. However, it is challenging to implement efficient input pipelines, as it requires reasoning about parallelism, asynchrony, and variability in fine-grained profiling information. Our analysis of over two million ML jobs in Google datacenters reveals that a significant fraction of model training jobs could benefit from faster input data pipelines. At the same time, our analysis indicates that most jobs do not saturate host hardware, pointing in the direction of software-based bottlenecks. Motivated by these findings, we propose Plumber, a tool for finding bottlenecks in ML input pipelines. Plumber uses an extensible and interpretable operational analysis analytical model to automatically tune parallelism, prefetching, and caching under host resource constraints. Across five representative ML pipelines, Plumber obtains speedups of up to 47x for misconfigured pipelines. By automating caching, Plumber obtains end-to-end speedups of over 50% compared to state-of-the-art tuners.","url_abs":"https://arxiv.org/abs/2111.04131v2","url_pdf":"https://arxiv.org/pdf/2111.04131v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"plumber-diagnosing-and-removing-performance","repo_url":"https://github.com/mkuchnik/plumberapp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"jax","reach":{"status":"ok"}},{"paper_slug":"plumber-diagnosing-and-removing-performance","repo_url":"https://github.com/mkuchnik/plumbertensorflow","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"machine-learning","task_name":"BIG-bench Machine Learning"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2111.04131","atlas_url":"https://app.syntology.ai/?focus=2111.04131","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2111.04131"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mkuchnik/plumberapp","reach":{"status":"ok"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mkuchnik/plumbertensorflow","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran_draft_wrong":2,"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":9,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"50ac80dd2372bb51","entry":"cygpath","repo":"mkuchnik/plumbertensorflow","repo_kind":"official","path":"configure.py","file_url":"https://github.com/mkuchnik/plumbertensorflow/blob/HEAD/configure.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"50ac80dd2372bb51"}},{"code_sha256_prefix":"e07e303581bd4892","entry":"get_input","repo":"mkuchnik/plumbertensorflow","repo_kind":"official","path":"configure.py","file_url":"https://github.com/mkuchnik/plumbertensorflow/blob/HEAD/configure.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e07e303581bd4892"}},{"code_sha256_prefix":"0031aaf5d0b6eef0","entry":"get_inputs_outputs","repo":"mkuchnik/plumbertensorflow","repo_kind":"official","path":"tensorflow/lite/python/convert_saved_model.py","file_url":"https://github.com/mkuchnik/plumbertensorflow/blob/HEAD/tensorflow/lite/python/convert_saved_model.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"0031aaf5d0b6eef0"}},{"code_sha256_prefix":"2034af57f4bcd949","entry":"minimize_loss_example","repo":"mkuchnik/plumbertensorflow","repo_kind":"official","path":"tensorflow/python/distribute/single_loss_example.py","file_url":"https://github.com/mkuchnik/plumbertensorflow/blob/HEAD/tensorflow/python/distribute/single_loss_example.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2034af57f4bcd949"}},{"code_sha256_prefix":"8ebf21a48f9d476d","entry":"run_shell","repo":"mkuchnik/plumbertensorflow","repo_kind":"official","path":"configure.py","file_url":"https://github.com/mkuchnik/plumbertensorflow/blob/HEAD/configure.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8ebf21a48f9d476d"}},{"code_sha256_prefix":"ce79211ba810feb6","entry":"single_loss_example","repo":"mkuchnik/plumbertensorflow","repo_kind":"official","path":"tensorflow/python/distribute/single_loss_example.py","file_url":"https://github.com/mkuchnik/plumbertensorflow/blob/HEAD/tensorflow/python/distribute/single_loss_example.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ce79211ba810feb6"}},{"code_sha256_prefix":"2cd4fe48e03992fc","entry":"batchnorm_example","repo":"mkuchnik/plumbertensorflow","repo_kind":"official","path":"tensorflow/python/distribute/single_loss_example.py","file_url":"https://github.com/mkuchnik/plumbertensorflow/blob/HEAD/tensorflow/python/distribute/single_loss_example.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2cd4fe48e03992fc"}},{"code_sha256_prefix":"9a832e63833396e4","entry":"get_meta_graph_def","repo":"mkuchnik/plumbertensorflow","repo_kind":"official","path":"tensorflow/lite/python/convert_saved_model.py","file_url":"https://github.com/mkuchnik/plumbertensorflow/blob/HEAD/tensorflow/lite/python/convert_saved_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a832e63833396e4"}},{"code_sha256_prefix":"1a24b33d13309a22","entry":"get_signature_def","repo":"mkuchnik/plumbertensorflow","repo_kind":"official","path":"tensorflow/lite/python/convert_saved_model.py","file_url":"https://github.com/mkuchnik/plumbertensorflow/blob/HEAD/tensorflow/lite/python/convert_saved_model.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1a24b33d13309a22"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}