{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-15286","title":"No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage","arxiv_id":"2608.15286","date":"2026-08-15","proceeding":null,"authors":["Shiven Khurdi"],"abstract":"We introduce AgentRelBench, an environment-agnostic reliability instrument that computes ground-truth, severity-priced damage from database state diffs across repeated runs, with no LLM in the measurement path, demonstrated on EnterpriseOps-Gym. Across 2,128 evaluation runs spanning nine models in six families (four development, three pre-registered held-out, plus a frontier pass on two frontier-tier models that the pre-registration designates exploratory), we find: (1) damage on irreversible actions is universal across the families we measured and stochastic within them on pinned, single-provider stacks. (2) No task damaged on every run: zero always-fail cells across 42 confirmatory held-out damage events. A single clean run misses a damage-producing (model, task) pair 0.80 of the time on the development pool (13 pairs); the held-out pool is descriptively consistent (0.575 over 5 pairs, pair-weighted) but sits below our pre-registered power floor and is reported as underpowered, not as confirmation. (3) Damage-producing task count falls with model capability, from 7 of 20 tasks for an 8B model to 1 of 20 for the most capable; capability is confounded with family and training, so this is an observed gradient, not a causal claim. The residual damage does not change in character: in the exploratory frontier pass, the most capable model's one damaging task damages at $\\hat{p} = 0.16$ per run, inside the same demonstrably-stochastic band, and a single audit misses it 84% of the time. (4) One model family committed the gated irreversible change while declaring it had refused: transcript- and judge-based grading scores those runs as safe refusals, only state diffs as damage. All confirmatory findings were pre-registered with per-claim demote criteria; one demoted our own initially favored finding, which we report.","url_abs":"https://arxiv.org/abs/2608.15286","url_pdf":"https://arxiv.org/pdf/2608.15286","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":null,"mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.15286"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/shivenkk/agentrelbench","reach":null}],"summary":{"ran":9,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":11,"ran":9,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"4b314c288935e52c","entry":"classify","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"m1_audit/test1_tool_inventory.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/m1_audit/test1_tool_inventory.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"4b314c288935e52c"}},{"code_sha256_prefix":"3e3fa5c13c323424","entry":"compare_verdict","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"src/agentrelbench/validate.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/src/agentrelbench/validate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3e3fa5c13c323424"}},{"code_sha256_prefix":"cd338b9953f70ef6","entry":"context_to_headers","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"src/agentrelbench/state_export.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/src/agentrelbench/state_export.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"cd338b9953f70ef6"}},{"code_sha256_prefix":"1190796a4f647c42","entry":"pass_pow_k","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"src/agentrelbench/estimators.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/src/agentrelbench/estimators.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"1190796a4f647c42"}},{"code_sha256_prefix":"149b4ee9fe1fce1a","entry":"per_task_stats","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"src/agentrelbench/estimators.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/src/agentrelbench/estimators.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"149b4ee9fe1fce1a"}},{"code_sha256_prefix":"be96dbbc35a3ccfd","entry":"redact_llm_config","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"src/agentrelbench/collector.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/src/agentrelbench/collector.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"be96dbbc35a3ccfd"}},{"code_sha256_prefix":"bf8f93f44a109e0e","entry":"safe_pow_k","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"src/agentrelbench/estimators.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/src/agentrelbench/estimators.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"bf8f93f44a109e0e"}},{"code_sha256_prefix":"9591dc1835c52a65","entry":"sampling_params","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"src/agentrelbench/collector.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/src/agentrelbench/collector.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9591dc1835c52a65"}},{"code_sha256_prefix":"d88835e538dad227","entry":"summarize","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"src/agentrelbench/labeler.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/src/agentrelbench/labeler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d88835e538dad227"}},{"code_sha256_prefix":"b401e33d37927494","entry":"main","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"src/agentrelbench/inner_runner.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/src/agentrelbench/inner_runner.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b401e33d37927494"}},{"code_sha256_prefix":"b8c70f2ac8dd54f2","entry":"mcp_image_digest","repo":"shivenkk/agentrelbench","repo_kind":"found_in_text","path":"src/agentrelbench/collector.py","file_url":"https://github.com/shivenkk/agentrelbench/blob/HEAD/src/agentrelbench/collector.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b8c70f2ac8dd54f2"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}