{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/executable-code-actions-elicit-better-llm","title":"Executable Code Actions Elicit Better LLM Agents","arxiv_id":"2402.01030","date":"2024-02-01","proceeding":null,"authors":["Xingyao Wang","Yangyi Chen","Lifan Yuan","Yizhe Zhang","Yunzhu Li","Hao Peng","Heng Ji"],"abstract":"Large Language Model (LLM) agents, capable of performing a broad range of actions, such as invoking tools and controlling robots, show great potential in tackling real-world challenges. LLM agents are typically prompted to produce actions by generating JSON or text in a pre-defined format, which is usually limited by constrained action space (e.g., the scope of pre-defined tools) and restricted flexibility (e.g., inability to compose multiple tools). This work proposes to use executable Python code to consolidate LLM agents' actions into a unified action space (CodeAct). Integrated with a Python interpreter, CodeAct can execute code actions and dynamically revise prior actions or emit new actions upon new observations through multi-turn interactions. Our extensive analysis of 17 LLMs on API-Bank and a newly curated benchmark shows that CodeAct outperforms widely used alternatives (up to 20% higher success rate). The encouraging performance of CodeAct motivates us to build an open-source LLM agent that interacts with environments by executing interpretable code and collaborates with users using natural language. To this end, we collect an instruction-tuning dataset CodeActInstruct that consists of 7k multi-turn interactions using CodeAct. We show that it can be used with existing data to improve models in agent-oriented tasks without compromising their general capability. CodeActAgent, finetuned from Llama2 and Mistral, is integrated with Python interpreter and uniquely tailored to perform sophisticated tasks (e.g., model training) using existing libraries and autonomously self-debug.","url_abs":"https://arxiv.org/abs/2402.01030v4","url_pdf":"https://arxiv.org/pdf/2402.01030v4.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"executable-code-actions-elicit-better-llm","repo_url":"https://github.com/epfllm/megatron-llm","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"NOASSERTION"}},{"paper_slug":"executable-code-actions-elicit-better-llm","repo_url":"https://github.com/xingyaoww/code-act","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"none","reach":null}],"tasks":[{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.01030","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.01030"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/epfllm/megatron-llm","reach":{"status":"ok","spdx":"NOASSERTION"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/xingyaoww/code-act","reach":null}],"summary":{"ran_violates":2,"ran":3,"ran_draft_wrong":1},"by_repo_kind":{"official":{"samples":3,"ran":3,"repositories":1},"community":{"samples":3,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"084846ce6b55992b","entry":"add","repo":"langchain-ai/langgraph-codeact","repo_kind":"community","path":"examples/math_example.py","file_url":"https://github.com/langchain-ai/langgraph-codeact/blob/HEAD/examples/math_example.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"084846ce6b55992b"}},{"code_sha256_prefix":"9cc6bf4a184ce3c1","entry":"multiply","repo":"langchain-ai/langgraph-codeact","repo_kind":"community","path":"examples/math_example.py","file_url":"https://github.com/langchain-ai/langgraph-codeact/blob/HEAD/examples/math_example.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":2,"contract_check":"VIOLATES","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9cc6bf4a184ce3c1"}},{"code_sha256_prefix":"cb65551e188ba525","entry":"Action","repo":"xingyaoww/code-act","repo_kind":"official","path":"mint/agents/base.py","file_url":"https://github.com/xingyaoww/code-act/blob/HEAD/mint/agents/base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cb65551e188ba525"}},{"code_sha256_prefix":"8b3b626b2c7c8c9c","entry":"LMAgent","repo":"xingyaoww/code-act","repo_kind":"official","path":"mint/agents/base.py","file_url":"https://github.com/xingyaoww/code-act/blob/HEAD/mint/agents/base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b3b626b2c7c8c9c"}},{"code_sha256_prefix":"bb465719419c1b62","entry":"State","repo":"xingyaoww/code-act","repo_kind":"official","path":"mint/agents/base.py","file_url":"https://github.com/xingyaoww/code-act/blob/HEAD/mint/agents/base.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bb465719419c1b62"}},{"code_sha256_prefix":"0eec38f3e939db61","entry":"eval","repo":"langchain-ai/langgraph-codeact","repo_kind":"community","path":"examples/math_example.py","file_url":"https://github.com/langchain-ai/langgraph-codeact/blob/HEAD/examples/math_example.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0eec38f3e939db61"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}