{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-28281","title":"LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering","arxiv_id":"2608.28281","date":"2026-08-28","proceeding":null,"authors":["Yi Wang","Haopeng Zhang","Chengxiang Huang","Rui Dai","Kaikui Liu","Piotr Koniusz","Xiangxiang Chu"],"abstract":"Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the \\textbf{Controller}: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the \\textbf{Worker}, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I scores next-step Loop Contract selection through execution-validated questions without running the Worker at evaluation time. Type II executes repeated control over a selected slice of a full task, while Type III evaluates the paired full task from its original state. On full tasks, the best observed Strict Success Rate is \\textbf{24.69\\%}, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages \\textbf{64.4\\%}, and Type II produces a similar ordering under the main Core criterion (Spearman's \\(ρ=\\textbf{0.9747}\\)). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena .","url_abs":"https://arxiv.org/abs/2608.28281","url_pdf":"https://arxiv.org/pdf/2608.28281","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2608.28281","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.28281"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/AMAP-ML/LoopArena","reach":null}],"summary":{"ran":8,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":12,"ran":8,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"2d2e8d3b531bf5da","entry":"acquire_panel_writer_lease","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/commands/type2_panel.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/commands/type2_panel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2d2e8d3b531bf5da"}},{"code_sha256_prefix":"a931e05553505f6b","entry":"jobs","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/commands/type3_panel.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/commands/type3_panel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a931e05553505f6b"}},{"code_sha256_prefix":"94737b3d35ed3146","entry":"localize_evaluator_plan","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/benchmarks/type3_runtime.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/benchmarks/type3_runtime.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"94737b3d35ed3146"}},{"code_sha256_prefix":"3f78f7bcd3deb220","entry":"localize_evaluator_plan","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/commands/type2_run.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/commands/type2_run.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3f78f7bcd3deb220"}},{"code_sha256_prefix":"a46c7e977f48c70c","entry":"panel_writer_active","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/commands/type2_panel.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/commands/type2_panel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a46c7e977f48c70c"}},{"code_sha256_prefix":"68131a46264551ab","entry":"read_object","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/benchmarks/type3.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/benchmarks/type3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68131a46264551ab"}},{"code_sha256_prefix":"2cc7ab3ef7219f62","entry":"run_dir","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/commands/type3_panel.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/commands/type3_panel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2cc7ab3ef7219f62"}},{"code_sha256_prefix":"cb6bfce53ed0b6d6","entry":"safe_relative","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/benchmarks/type3.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/benchmarks/type3.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cb6bfce53ed0b6d6"}},{"code_sha256_prefix":"63d8975575b4f621","entry":"preflight_type2_case","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/commands/type2_run.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/commands/type2_run.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"63d8975575b4f621"}},{"code_sha256_prefix":"a8c21f6c4ada713e","entry":"sha256_file","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/benchmarks/type3.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/benchmarks/type3.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a8c21f6c4ada713e"}},{"code_sha256_prefix":"63a309bcd57f8929","entry":"summarize_release","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/commands/results_summarize.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/commands/results_summarize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"63a309bcd57f8929"}},{"code_sha256_prefix":"b316e927dce3ae6a","entry":"to_markdown","repo":"AMAP-ML/LoopArena","repo_kind":"found_in_text","path":"src/looparena/commands/type2_summarize.py","file_url":"https://github.com/AMAP-ML/LoopArena/blob/HEAD/src/looparena/commands/type2_summarize.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b316e927dce3ae6a"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.AI","source":"arxiv_daily_20260831.json"},"syntology_extracted_results":null}