{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-03691","title":"Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality","arxiv_id":"2607.03691","date":"2026-07-04","proceeding":null,"authors":["Oussama Ben Sghaier","Hao Li","Bram Adams","Ahmed E. Hassan"],"abstract":"Coding agents, autonomous systems that use large language models (LLMs) to resolve software engineering tasks, rely on agent harness: a middleware layer in between a developer and a large language model that orchestrates system prompts, tool execution, context management, and iterative reasoning loops. While these agent harnesses evolve at extreme velocities, no study has examined how this evolution affects agent quality (i.e., effectiveness and efficiency) over time. Practitioners regularly report quality regressions after agent harness updates, yet consistently attribute them to the underlying model rather than the harness itself. In this paper, we address this gap by conducting the first controlled longitudinal study that isolates the agent harness contribution. Unlike prior work that fixes the agent harness and varies the model, we fix the model and vary only the agent harness, evaluating 35 sequential releases to measure their impact on agent effectiveness and efficiency. We first empirically study the development and release evolution of five major open-source agent harnesses (i.e., Codex, Qwen Code, Gemini, OpenCode, and OpenHands), revealing extreme release velocities exceeding two releases per day and thousands of issues within months. We then perform a controlled deep dive into 35 sequential releases of the Qwen Code CLI, evaluating each against 50 stratified SWE-bench Verified tasks while holding the underlying LLM constant. We trace the resulting quality fluctuations to specific development patterns and architectural components, and illustrate our findings with concrete qualitative evidence linking individual pull requests to measured quality shifts.","url_abs":"https://arxiv.org/abs/2607.03691","url_pdf":"https://arxiv.org/pdf/2607.03691","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2607.03691","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.03691"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/QwenLM/qwen-code","reach":null}],"summary":{"ran":7,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":8,"ran":7,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4c235acc0bad6c5a","entry":"get_platform_info","repo":"QwenLM/qwen-code","repo_kind":"found_in_text","path":"packages/cua-driver/python/build_wheel.py","file_url":"https://github.com/QwenLM/qwen-code/blob/HEAD/packages/cua-driver/python/build_wheel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4c235acc0bad6c5a"}},{"code_sha256_prefix":"15441672d42f0eda","entry":"get_release_url","repo":"QwenLM/qwen-code","repo_kind":"found_in_text","path":"packages/cua-driver/python/build_wheel.py","file_url":"https://github.com/QwenLM/qwen-code/blob/HEAD/packages/cua-driver/python/build_wheel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"15441672d42f0eda"}},{"code_sha256_prefix":"1210220744c016ee","entry":"get_wheel_tag","repo":"QwenLM/qwen-code","repo_kind":"found_in_text","path":"packages/cua-driver/python/build_wheel.py","file_url":"https://github.com/QwenLM/qwen-code/blob/HEAD/packages/cua-driver/python/build_wheel.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1210220744c016ee"}},{"code_sha256_prefix":"27ae4a59ed3d9f40","entry":"is_sdk_system_message","repo":"QwenLM/qwen-code","repo_kind":"found_in_text","path":"packages/sdk-python/src/qwen_code_sdk/protocol.py","file_url":"https://github.com/QwenLM/qwen-code/blob/HEAD/packages/sdk-python/src/qwen_code_sdk/protocol.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"27ae4a59ed3d9f40"}},{"code_sha256_prefix":"a387c66349a27a0f","entry":"parse_json_line","repo":"QwenLM/qwen-code","repo_kind":"found_in_text","path":"packages/sdk-python/src/qwen_code_sdk/json_lines.py","file_url":"https://github.com/QwenLM/qwen-code/blob/HEAD/packages/sdk-python/src/qwen_code_sdk/json_lines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"a387c66349a27a0f"}},{"code_sha256_prefix":"ddcebcc37b304c92","entry":"prepare_spawn_info","repo":"QwenLM/qwen-code","repo_kind":"found_in_text","path":"packages/sdk-python/src/qwen_code_sdk/transport.py","file_url":"https://github.com/QwenLM/qwen-code/blob/HEAD/packages/sdk-python/src/qwen_code_sdk/transport.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ddcebcc37b304c92"}},{"code_sha256_prefix":"eae0b2b7130c9242","entry":"serialize_json_line","repo":"QwenLM/qwen-code","repo_kind":"found_in_text","path":"packages/sdk-python/src/qwen_code_sdk/json_lines.py","file_url":"https://github.com/QwenLM/qwen-code/blob/HEAD/packages/sdk-python/src/qwen_code_sdk/json_lines.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eae0b2b7130c9242"}},{"code_sha256_prefix":"68a77147ac3b63f9","entry":"load_config","repo":"QwenLM/qwen-code","repo_kind":"found_in_text","path":"integration-tests/concurrent-runner/runner.py","file_url":"https://github.com/QwenLM/qwen-code/blob/HEAD/integration-tests/concurrent-runner/runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"68a77147ac3b63f9"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}