{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/textatari-100k-frames-game-playing-with","title":"TextAtari: 100K Frames Game Playing with Language Agents","arxiv_id":"2506.04098","date":"2025-06-04","proceeding":null,"authors":["Wenhao Li","Wenwu Li","Chuyun Shen","Junjie Sheng","Zixiao Huang","Di wu","Yun Hua","Wei Yin","Xiangfeng Wang","Hongyuan Zha","Bo Jin"],"abstract":"We present TextAtari, a benchmark for evaluating language agents on very long-horizon decision-making tasks spanning up to 100,000 steps. By translating the visual state representations of classic Atari games into rich textual descriptions, TextAtari creates a challenging test bed that bridges sequential decision-making with natural language processing. The benchmark includes nearly 100 distinct tasks with varying complexity, action spaces, and planning horizons, all rendered as text through an unsupervised representation learning framework (AtariARI). We evaluate three open-source large language models (Qwen2.5-7B, Gemma-7B, and Llama3.1-8B) across three agent frameworks (zero-shot, few-shot chain-of-thought, and reflection reasoning) to assess how different forms of prior knowledge affect performance on these long-horizon challenges. Four scenarios-Basic, Obscured, Manual Augmentation, and Reference-based-investigate the impact of semantic understanding, instruction comprehension, and expert demonstrations on agent decision-making. Our results reveal significant performance gaps between language agents and human players in extensive planning tasks, highlighting challenges in sequential reasoning, state tracking, and strategic planning across tens of thousands of steps. TextAtari provides standardized evaluation protocols, baseline implementations, and a framework for advancing research at the intersection of language models and planning.","url_abs":"https://arxiv.org/abs/2506.04098v1","url_pdf":"https://arxiv.org/pdf/2506.04098v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"textatari-100k-frames-game-playing-with","repo_url":"https://github.com/Lww007/Text-Atari-Agents","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"none","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"atari-games","task_name":"Atari Games"},{"task_slug":"decision-making","task_name":"Decision Making"},{"task_slug":"representation-learning","task_name":"Representation Learning"},{"task_slug":"sequential-decision-making","task_name":"Sequential Decision Making"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2506.04098","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2506.04098"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Lww007/Text-Atari-Agents","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":5},"by_repo_kind":{"official":{"samples":5,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"c9f4caaff330b5e6","entry":"dedicated_1_policy","repo":"Lww007/Text-Atari-Agents","repo_kind":"official","path":"envs/atari/Bowling_policies.py","file_url":"https://github.com/Lww007/Text-Atari-Agents/blob/HEAD/envs/atari/Bowling_policies.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c9f4caaff330b5e6"}},{"code_sha256_prefix":"92224ce167966663","entry":"dedicated_2_policy","repo":"Lww007/Text-Atari-Agents","repo_kind":"official","path":"envs/atari/Bowling_policies.py","file_url":"https://github.com/Lww007/Text-Atari-Agents/blob/HEAD/envs/atari/Bowling_policies.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"92224ce167966663"}},{"code_sha256_prefix":"89b7178d8cf1d7c5","entry":"dedicated_2_policy","repo":"Lww007/Text-Atari-Agents","repo_kind":"official","path":"envs/atari/Boxing_policies.py","file_url":"https://github.com/Lww007/Text-Atari-Agents/blob/HEAD/envs/atari/Boxing_policies.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"89b7178d8cf1d7c5"}},{"code_sha256_prefix":"cc9c17a9b49ba300","entry":"dedicated_3_policy","repo":"Lww007/Text-Atari-Agents","repo_kind":"official","path":"envs/atari/Bowling_policies.py","file_url":"https://github.com/Lww007/Text-Atari-Agents/blob/HEAD/envs/atari/Bowling_policies.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cc9c17a9b49ba300"}},{"code_sha256_prefix":"17733e46541fcd98","entry":"dedicated_3_policy","repo":"Lww007/Text-Atari-Agents","repo_kind":"official","path":"envs/atari/Boxing_policies.py","file_url":"https://github.com/Lww007/Text-Atari-Agents/blob/HEAD/envs/atari/Boxing_policies.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"17733e46541fcd98"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}