{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-04202","title":"ClawArena: Benchmarking AI Agents in Evolving Information Environments","arxiv_id":"2604.04202","date":"2026-04-05","proceeding":null,"authors":["Haonian Ji","Kaiwen Xiong","Siwei Han","Peng Xia","Shi Qiu","Yiyang Zhou","Jiaqi Liu","Jinlong Li","Bingzhou Li","Zeyu Zheng","Cihang Xie","Huaxiu Yao"],"abstract":"AI agents deployed as persistent assistants must maintain correct beliefs as their information environment evolves. In practice, evidence is scattered across heterogeneous sources that often contradict one another, new information can invalidate earlier conclusions, and user preferences surface through corrections rather than explicit instructions. Existing benchmarks largely assume static, single-authority settings and do not evaluate whether agents can keep up with this complexity. We introduce ClawArena, a benchmark for evaluating AI agents in evolving information environments. Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces across multi-channel sessions, workspace files, and staged updates. Evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization, whose interactions yield a 14-category question taxonomy. Two question formats, multi-choice (set-selection) and shell-based executable checks, test both reasoning and workspace grounding. ClawArena comprises 12 multi-turn scenarios spanning 337 evaluation rounds with 45 dynamic updates, evaluated across five agent frameworks and 18 language models from proprietary, community-accessible, and self-hosted sources. Experiments show that model capability accounts for a 29-point score range across models while framework design accounts for up to a 24-point range, that MetaClaw's skill overlay reliably improves score without degrading accuracy, and that belief revision difficulty is determined by update design strategy rather than update volume. Code is available at https://github.com/aiming-lab/ClawArena.","url_abs":"https://arxiv.org/abs/2604.04202","url_pdf":"https://arxiv.org/pdf/2604.04202","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2604.04202","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.04202"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/aiming-lab/ClawArena","reach":null}],"summary":{"ran":3,"ran_honours":4,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6cd1d56be50b6b11","entry":"Modality","repo":"aiming-lab/ClawArena","repo_kind":"found_in_text","path":"ClawArena-Team/src/clawarena_team/scoring/metrics.py","file_url":"https://github.com/aiming-lab/ClawArena/blob/HEAD/ClawArena-Team/src/clawarena_team/scoring/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6cd1d56be50b6b11"}},{"code_sha256_prefix":"a934ef44cbf8f0f6","entry":"RoundEval","repo":"aiming-lab/ClawArena","repo_kind":"found_in_text","path":"ClawArena-Team/src/clawarena_team/scoring/metrics.py","file_url":"https://github.com/aiming-lab/ClawArena/blob/HEAD/ClawArena-Team/src/clawarena_team/scoring/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a934ef44cbf8f0f6"}},{"code_sha256_prefix":"bc69614ef768171b","entry":"ScenarioMetrics","repo":"aiming-lab/ClawArena","repo_kind":"found_in_text","path":"ClawArena-Team/src/clawarena_team/scoring/metrics.py","file_url":"https://github.com/aiming-lab/ClawArena/blob/HEAD/ClawArena-Team/src/clawarena_team/scoring/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc69614ef768171b"}},{"code_sha256_prefix":"dad035c725e86476","entry":"_model_choice_score","repo":"aiming-lab/ClawArena","repo_kind":"found_in_text","path":"ClawArena-Team/src/clawarena_team/scoring/metrics.py","file_url":"https://github.com/aiming-lab/ClawArena/blob/HEAD/ClawArena-Team/src/clawarena_team/scoring/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"dad035c725e86476"}},{"code_sha256_prefix":"00733b56680c5e87","entry":"_readonly_score","repo":"aiming-lab/ClawArena","repo_kind":"found_in_text","path":"ClawArena-Team/src/clawarena_team/scoring/metrics.py","file_url":"https://github.com/aiming-lab/ClawArena/blob/HEAD/ClawArena-Team/src/clawarena_team/scoring/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00733b56680c5e87"}},{"code_sha256_prefix":"7e5e0ea71970628e","entry":"_tool_permission_score","repo":"aiming-lab/ClawArena","repo_kind":"found_in_text","path":"ClawArena-Team/src/clawarena_team/scoring/metrics.py","file_url":"https://github.com/aiming-lab/ClawArena/blob/HEAD/ClawArena-Team/src/clawarena_team/scoring/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7e5e0ea71970628e"}},{"code_sha256_prefix":"06a6bc82eb7b8691","entry":"_workspace_score","repo":"aiming-lab/ClawArena","repo_kind":"found_in_text","path":"ClawArena-Team/src/clawarena_team/scoring/metrics.py","file_url":"https://github.com/aiming-lab/ClawArena/blob/HEAD/ClawArena-Team/src/clawarena_team/scoring/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"06a6bc82eb7b8691"}},{"code_sha256_prefix":"8242effbc42eb537","entry":"compute_scenario_metrics","repo":"aiming-lab/ClawArena","repo_kind":"found_in_text","path":"ClawArena-Team/src/clawarena_team/scoring/metrics.py","file_url":"https://github.com/aiming-lab/ClawArena/blob/HEAD/ClawArena-Team/src/clawarena_team/scoring/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8242effbc42eb537"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}