{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2602-11103","title":"GameDevBench: Evaluating Agentic Capabilities Through Game Development","arxiv_id":"2602.11103","date":"2026-02-11","proceeding":"ICML","authors":["Wayne Chi","Yixiong Fang","Arnav Yayavaram","Siddharth Yayavaram","Seth Karten","Qiuhong Anna Wei","Runkun Chen","Alexander Wang","Valerie Chen","Ameet Talwalkar","Chris Donahue"],"abstract":"Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. In game development, agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 333 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex: the average solution requires over three times the lines of code and file changes compared to prior software development benchmarks. Agents struggle with game development, with the best agent and method solving only 53.8% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with average success rate dropping from 51.4% on gameplay-oriented tasks to 33.0% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image- and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, increasing GPT-5.4's performance from 41.1% to 52.0% when given visual feedback.","url_abs":"https://arxiv.org/abs/2602.11103","url_pdf":"https://arxiv.org/pdf/2602.11103","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2602.11103","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2602.11103"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/jdepoix/youtube-transcript-api","reach":{"status":"ok","spdx":"MIT"}},{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/waynchi/gamedevbench","reach":null}],"summary":{"unverified":4},"by_repo_kind":{"found_in_text":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8933e9caadf04164","entry":"compress_screenshot","repo":"waynchi/gamedevbench","repo_kind":"found_in_text","path":"gamedevbench/src/mcp_server.py","file_url":"https://github.com/waynchi/gamedevbench/blob/HEAD/gamedevbench/src/mcp_server.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8933e9caadf04164"}},{"code_sha256_prefix":"257cd40bfce458cc","entry":"host_is_allowed","repo":"waynchi/gamedevbench","repo_kind":"found_in_text","path":"gamedevbench/src/provider_proxy.py","file_url":"https://github.com/waynchi/gamedevbench/blob/HEAD/gamedevbench/src/provider_proxy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"257cd40bfce458cc"}},{"code_sha256_prefix":"d5cf9cdc8cc36413","entry":"normalize_host","repo":"waynchi/gamedevbench","repo_kind":"found_in_text","path":"gamedevbench/src/provider_proxy.py","file_url":"https://github.com/waynchi/gamedevbench/blob/HEAD/gamedevbench/src/provider_proxy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d5cf9cdc8cc36413"}},{"code_sha256_prefix":"231f5e66ae7d260e","entry":"run_namespace_relay","repo":"waynchi/gamedevbench","repo_kind":"found_in_text","path":"gamedevbench/src/provider_proxy.py","file_url":"https://github.com/waynchi/gamedevbench/blob/HEAD/gamedevbench/src/provider_proxy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"231f5e66ae7d260e"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}