{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-15715","title":"GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows","arxiv_id":"2604.15715","date":"2026-04-17","proceeding":null,"authors":["Jize Wang","Xuanxuan Liu","Yining Li","Songyang Zhang","Yijun Wang","Zifei Shan","Xinyi Le","Cailian Chen","Xinping Guan","Dacheng Tao"],"abstract":"The development of general-purpose agents requires a shift from executing simple instructions to completing complex, real-world productivity workflows. However, current tool-use benchmarks remain misaligned with real-world requirements, relying on AI-generated queries, dummy tools, and limited system-level coordination. To address this, we propose GTA-2, a hierarchical benchmark for General Tool Agents (GTA) spanning atomic tool use and open-ended workflows. Built on real-world authenticity, it leverages real user queries, deployed tools, and multimodal contexts. (i) GTA-Atomic, inherited from our prior GTA benchmark, evaluates short-horizon, closed-ended tool-use precision. (ii) GTA-Workflow introduces long-horizon, open-ended tasks for realistic end-to-end completion. To evaluate open-ended deliverables, we propose a recursive checkpoint-based evaluation mechanism that decomposes objectives into verifiable sub-goals, enabling unified evaluation of both model capabilities and agent execution frameworks (i.e., execution harnesses). Experiments reveal a pronounced capability cliff: while frontier models already struggle on atomic tasks (below 50%), they largely fail on workflows, with top models achieving only 14.39% success. Further analysis shows that checkpoint-guided feedback improves performance, while advanced frameworks such as Manus and OpenClaw substantially enhance workflow completion, highlighting the importance of execution harness design beyond the underlying model capacity. These findings provide guidance for developing reliable personal and professional assistants. Dataset and code will be available at https://github.com/open-compass/GTA.","url_abs":"https://arxiv.org/abs/2604.15715","url_pdf":"https://arxiv.org/pdf/2604.15715","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2604.15715","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.15715"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/open-compass/GTA","reach":null}],"summary":{"unverified":7},"by_repo_kind":{"found_in_text":{"samples":7,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"1db4860160f2ef07","entry":"build_eval_pack","repo":"open-compass/GTA","repo_kind":"found_in_text","path":"agent_app_eval/run_agents.py","file_url":"https://github.com/open-compass/GTA/blob/HEAD/agent_app_eval/run_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1db4860160f2ef07"}},{"code_sha256_prefix":"8c3143cad4a6cd5a","entry":"generate_readme","repo":"open-compass/GTA","repo_kind":"found_in_text","path":"agentlego/.dev_scripts/generate_readme.py","file_url":"https://github.com/open-compass/GTA/blob/HEAD/agentlego/.dev_scripts/generate_readme.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8c3143cad4a6cd5a"}},{"code_sha256_prefix":"bb767be06d938e9b","entry":"get_all_file_paths","repo":"open-compass/GTA","repo_kind":"found_in_text","path":"opencompass/opencompass/datasets/gta_bench.py","file_url":"https://github.com/open-compass/GTA/blob/HEAD/opencompass/opencompass/datasets/gta_bench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bb767be06d938e9b"}},{"code_sha256_prefix":"bdfa48481af8eb52","entry":"get_all_file_paths","repo":"open-compass/GTA","repo_kind":"found_in_text","path":"opencompass/opencompass/datasets/gta_bench_v2.py","file_url":"https://github.com/open-compass/GTA/blob/HEAD/opencompass/opencompass/datasets/gta_bench_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bdfa48481af8eb52"}},{"code_sha256_prefix":"3ab02673548e9f4a","entry":"load_tasks","repo":"open-compass/GTA","repo_kind":"found_in_text","path":"agent_app_eval/run_agents.py","file_url":"https://github.com/open-compass/GTA/blob/HEAD/agent_app_eval/run_agents.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3ab02673548e9f4a"}},{"code_sha256_prefix":"4ccdff58f54c54fe","entry":"organize_dialogs","repo":"open-compass/GTA","repo_kind":"found_in_text","path":"opencompass/opencompass/datasets/gta_bench.py","file_url":"https://github.com/open-compass/GTA/blob/HEAD/opencompass/opencompass/datasets/gta_bench.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4ccdff58f54c54fe"}},{"code_sha256_prefix":"ccc6d1f0eb2cc097","entry":"organize_dialogs","repo":"open-compass/GTA","repo_kind":"found_in_text","path":"opencompass/opencompass/datasets/gta_bench_v2.py","file_url":"https://github.com/open-compass/GTA/blob/HEAD/opencompass/opencompass/datasets/gta_bench_v2.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ccc6d1f0eb2cc097"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}