{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/creative-agents-empowering-agents-with","title":"Creative Agents: Empowering Agents with Imagination for Creative Tasks","arxiv_id":"2312.02519","date":"2023-12-05","proceeding":null,"authors":["Chi Zhang","Penglin Cai","Yuhui Fu","Haoqi Yuan","Zongqing Lu"],"abstract":"We study building embodied agents for open-ended creative tasks. While existing methods build instruction-following agents that can perform diverse open-ended tasks, none of them demonstrates creativity -- the ability to give novel and diverse task solutions implicit in the language instructions. This limitation comes from their inability to convert abstract language instructions into concrete task goals in the environment and perform long-horizon planning for such complicated goals. Given the observation that humans perform creative tasks with the help of imagination, we propose a class of solutions for creative agents, where the controller is enhanced with an imaginator that generates detailed imaginations of task outcomes conditioned on language instructions. We introduce several approaches to implementing the components of creative agents. We implement the imaginator with either a large language model for textual imagination or a diffusion model for visual imagination. The controller can either be a behavior-cloning policy learned from data or a pre-trained foundation model generating executable codes in the environment. We benchmark creative tasks with the challenging open-world game Minecraft, where the agents are asked to create diverse buildings given free-form language instructions. In addition, we propose novel evaluation metrics for open-ended creative tasks utilizing GPT-4V, which holds many advantages over existing metrics. We perform a detailed experimental analysis of creative agents, showing that creative agents are the first AI agents accomplishing diverse building creation in the survival mode of Minecraft. Our benchmark and models are open-source for future research on creative agents (https://github.com/PKU-RL/Creative-Agents).","url_abs":"https://arxiv.org/abs/2312.02519v1","url_pdf":"https://arxiv.org/pdf/2312.02519v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"creative-agents-empowering-agents-with","repo_url":"https://github.com/pku-rl/creative-agents","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"jax","reach":{"status":"ok"}}],"tasks":[{"task_slug":"instruction-following","task_name":"Instruction Following"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"large-language-model","task_name":"Large Language Model"},{"task_slug":"minecraft","task_name":"Minecraft"}],"methods":[{"method_slug":"diffusion","method_name":"Diffusion"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2312.02519","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2312.02519"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/pku-rl/creative-agents","reach":{"status":"ok"}}],"summary":{"ran":2,"ran_draft_wrong":2,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"b1c2838ce19a2767","entry":"block2num","repo":"pku-rl/creative-agents","repo_kind":"official","path":"block2num.py","file_url":"https://github.com/pku-rl/creative-agents/blob/HEAD/block2num.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1c2838ce19a2767"}},{"code_sha256_prefix":"d3928f6429ce22ab","entry":"check","repo":"pku-rl/creative-agents","repo_kind":"official","path":"cot_gpt4.py","file_url":"https://github.com/pku-rl/creative-agents/blob/HEAD/cot_gpt4.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d3928f6429ce22ab"}},{"code_sha256_prefix":"213ccc8fb87ee8d9","entry":"conv1x1x1","repo":"pku-rl/creative-agents","repo_kind":"official","path":"BC_Controller/vox2seq/model.py","file_url":"https://github.com/pku-rl/creative-agents/blob/HEAD/BC_Controller/vox2seq/model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"213ccc8fb87ee8d9"}},{"code_sha256_prefix":"c3b215555357b9d1","entry":"conv3x3x3","repo":"pku-rl/creative-agents","repo_kind":"official","path":"BC_Controller/vox2seq/model.py","file_url":"https://github.com/pku-rl/creative-agents/blob/HEAD/BC_Controller/vox2seq/model.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c3b215555357b9d1"}},{"code_sha256_prefix":"53d35ee23e46b1d6","entry":"round1","repo":"pku-rl/creative-agents","repo_kind":"official","path":"cot_gpt4.py","file_url":"https://github.com/pku-rl/creative-agents/blob/HEAD/cot_gpt4.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"53d35ee23e46b1d6"}},{"code_sha256_prefix":"635257e85629c12b","entry":"round1","repo":"pku-rl/creative-agents","repo_kind":"official","path":"diffusion_gpt4v.py","file_url":"https://github.com/pku-rl/creative-agents/blob/HEAD/diffusion_gpt4v.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"635257e85629c12b"}},{"code_sha256_prefix":"39fc09fc88a4749f","entry":"round2","repo":"pku-rl/creative-agents","repo_kind":"official","path":"cot_gpt4.py","file_url":"https://github.com/pku-rl/creative-agents/blob/HEAD/cot_gpt4.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"39fc09fc88a4749f"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}