{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-22808","title":"CatchBench: When Can an Agent Failure Be Caught?","arxiv_id":"2608.22808","date":"2026-08-24","proceeding":null,"authors":["Yue Zhao"],"abstract":"When can an agent failure be caught? An audit is usually limited by the record rather than by the method. CatchBench therefore puts one auditor's question to three information states: the declared configuration before a run (PRE), a growing prefix of its trace (LIVE), and the finished trace (POST). Prior benchmarks fix one of these states or vary the telemetry; to our knowledge none scores all three under one task-method interface. Each state admits different questions, so seven task contracts carry their own labels and metrics rather than one leaderboard. Four are evidential; three are Gold-derived mechanism diagnostics. The release scores 72 entrants, from rule scanners and structural models to eleven LLM judges across nine model families (GPT, Claude, Gemini, Gemma, Llama, Qwen, DeepSeek, Mistral, Nova), over 1187 declared configurations and 1162 recorded runs. Most of the arena does not order: 47 of 118 pre-declared contrasts separate, and the rest are published unresolved rather than ranked. The two sharpest results cut against our own data. One rule ignores every name and permission; it flags each capability declared after the first. On one of six configuration sources it reaches a perfect F1, so a score there measures how the corpus was built rather than how well a method reasons. Our admissibility bar then rejected one injected substrate and withheld evidential status from the other. A benchmark number is therefore not interpretable until the process behind its labels is published and tested for the shortcut it may leave. We report both, and regenerate every ordering from released predictions with no model call.","url_abs":"https://arxiv.org/abs/2608.22808","url_pdf":"https://arxiv.org/pdf/2608.22808","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2608.22808","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.22808"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/yzhao062/catchbench","reach":null}],"summary":{"ran":12,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":13,"ran":12,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0cb015f5af4138ae","entry":"add_composition_bar","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"figure-src/catchbench_data_at_a_glance.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/figure-src/catchbench_data_at_a_glance.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0cb015f5af4138ae"}},{"code_sha256_prefix":"379c4ceb9dcfd0f3","entry":"add_round_rect","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"figure-src/catchbench_data_at_a_glance.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/figure-src/catchbench_data_at_a_glance.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"379c4ceb9dcfd0f3"}},{"code_sha256_prefix":"b782520e431bfb87","entry":"all_at_once_prompt","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/llm_judge.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/llm_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b782520e431bfb87"}},{"code_sha256_prefix":"a61445ce725ddb19","entry":"data_dir","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/_paths.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/_paths.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a61445ce725ddb19"}},{"code_sha256_prefix":"bdf8d7729764a38a","entry":"flat_disconnected","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/graph_ad.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/graph_ad.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bdf8d7729764a38a"}},{"code_sha256_prefix":"bffcf184c2592e83","entry":"full_context_edges","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/graph_ad.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/graph_ad.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bffcf184c2592e83"}},{"code_sha256_prefix":"a8603cbce6e20ae5","entry":"gsafeguard_cv_auc","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/agent_detectors.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/agent_detectors.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a8603cbce6e20ae5"}},{"code_sha256_prefix":"eabab1ab5753a3de","entry":"guardian_run_scores","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/agent_detectors.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/agent_detectors.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eabab1ab5753a3de"}},{"code_sha256_prefix":"e7b1d87f0e41ecbe","entry":"parse_prediction","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/llm_judge.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/llm_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7b1d87f0e41ecbe"}},{"code_sha256_prefix":"bbf447a83ce4d773","entry":"revision_header","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/corpora.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/corpora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bbf447a83ce4d773"}},{"code_sha256_prefix":"643c3c4b0c08ae7a","entry":"run_key","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/llm_judge.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/llm_judge.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"643c3c4b0c08ae7a"}},{"code_sha256_prefix":"860f08f02a107d52","entry":"verify_corpus_heads","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/corpora.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/corpora.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"860f08f02a107d52"}},{"code_sha256_prefix":"560f5a635a5e285a","entry":"pygod_node_scores","repo":"yzhao062/catchbench","repo_kind":"found_in_text","path":"src/catchbench/graph_ad.py","file_url":"https://github.com/yzhao062/catchbench/blob/HEAD/src/catchbench/graph_ad.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"560f5a635a5e285a"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}