{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/the-vera-c-rubin-observatory-data-butler-and","title":"The Vera C. Rubin Observatory Data Butler and Pipeline Execution System","arxiv_id":"2206.14941","date":"2022-06-29","proceeding":null,"authors":["Tim Jenness","James F. Bosch","Nate B. Lust","Nathan M. Pease","Michelle Gower","Mikolaj Kowalik","Gregory P. Dubois-Felsmann","Fritz Mueller","Pim Schellart"],"abstract":"The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a laptop using a local file system. The Butler and pipeline system are now in daily use during Rubin construction and early operations.","url_abs":"https://arxiv.org/abs/2206.14941v1","url_pdf":"https://arxiv.org/pdf/2206.14941v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"links_only","authors_date_abstract":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license), from the Kaggle arXiv metadata snapshot of 2026-09-12"},"code_links":[{"paper_slug":"the-vera-c-rubin-observatory-data-butler-and","repo_url":"https://github.com/lsst/ctrl_mpexec","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"the-vera-c-rubin-observatory-data-butler-and","repo_url":"https://github.com/lsst/pex_config","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null},{"paper_slug":"the-vera-c-rubin-observatory-data-butler-and","repo_url":"https://github.com/lsst/daf_butler","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":null,"mcp":null,"developers":"https://syntology.ai/developers"},"arxiv_metadata":null,"syntology_extracted_results":null}