{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/curie-toward-rigorous-and-automated","title":"Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agents","arxiv_id":"2502.16069","date":"2025-02-22","proceeding":null,"authors":["Patrick Tser Jern Kon","Jiachen Liu","Qiuyi Ding","Yiming Qiu","Zhenning Yang","Yibo Huang","Jayanth Srinivasa","Myungjin Lee","Mosharaf Chowdhury","Ang Chen"],"abstract":"Scientific experimentation, a cornerstone of human progress, demands rigor in reliability, methodical control, and interpretability to yield meaningful results. Despite the growing capabilities of large language models (LLMs) in automating different aspects of the scientific process, automating rigorous experimentation remains a significant challenge. To address this gap, we propose Curie, an AI agent framework designed to embed rigor into the experimentation process through three key components: an intra-agent rigor module to enhance reliability, an inter-agent rigor module to maintain methodical control, and an experiment knowledge module to enhance interpretability. To evaluate Curie, we design a novel experimental benchmark composed of 46 questions across four computer science domains, derived from influential research papers, and widely adopted open-source projects. Compared to the strongest baseline tested, we achieve a 3.4$\\times$ improvement in correctly answering experimental questions.Curie is open-sourced at https://github.com/Just-Curieous/Curie.","url_abs":"https://arxiv.org/abs/2502.16069v1","url_pdf":"https://arxiv.org/pdf/2502.16069v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"curie-toward-rigorous-and-automated","repo_url":"https://github.com/just-curieous/curie","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"ai-agent","task_name":"AI Agent"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2502.16069","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2502.16069"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/just-curieous/curie","reach":null}],"summary":{"ran_violates":1,"unverified":2},"by_repo_kind":{"official":{"samples":3,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"7182f9e553560ee0","entry":"docker_image_exists","repo":"just-curieous/curie","repo_kind":"official","path":"curie/experiment.py","file_url":"https://github.com/just-curieous/curie/blob/HEAD/curie/experiment.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"7182f9e553560ee0"}},{"code_sha256_prefix":"8f28bc70f91ef05a","entry":"execute_experiment_in_container","repo":"just-curieous/curie","repo_kind":"official","path":"curie/experiment.py","file_url":"https://github.com/just-curieous/curie/blob/HEAD/curie/experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8f28bc70f91ef05a"}},{"code_sha256_prefix":"d66c38b718ce68e3","entry":"run_docker_container","repo":"just-curieous/curie","repo_kind":"official","path":"curie/experiment.py","file_url":"https://github.com/just-curieous/curie/blob/HEAD/curie/experiment.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d66c38b718ce68e3"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}