{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2603-10044","title":"Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety","arxiv_id":"2603.10044","date":"2026-03-08","proceeding":null,"authors":["David Gringras"],"abstract":"A safety score earned on a benchmark need not predict how the same model behaves once it is wrapped in an agentic scaffold the benchmark never tested. We ran six frontier models through four deployment configurations (direct API, ReAct, multi-agent critic, map-reduce delegation): N = 62,808 blinded, pre-registered, equivalence-tested evaluations across four safety benchmarks (BBQ, TruthfulQA, XSTest/OR-Bench, sycophancy), plus three supporting analyses. ReAct and multi-agent scaffolds stay within a pre-registered +/-2 pp equivalence margin; map-reduce delegation degrades measured safety (NNH = 14), though that loss is largely a measurement artifact: on identical items, multiple-choice versus open-ended phrasing shifts the measured safety rate by 5-20 pp, and decomposition silently strips the multiple-choice options. Roughly 40-89% of the per-model map-reduce loss is this format conversion rather than reasoning disruption, and an option-preserving variant recovers most of it. Pooled effects also mask sharp model-by-scaffold heterogeneity: under map-reduce, on identical items, Opus loses 16.8 pp while Llama 4 gains 18.8 pp. Structurally, scaffold architecture explains only 0.4% of outcome variance (benchmark choice explains 45x more), and the generalizability coefficient is G = 0.000 (bootstrap 95% CI [0.000, 0.752]). An interval that wide is enough on its own to undermine the utility of any single composite safety number as a deployment criterion. These are the \"easy cases\"; consequential properties like scheming and CBRN uplift have no obvious reason to be less format- or scaffold-sensitive. Code, data, and prompts are released as ScaffoldSafety.","url_abs":"https://arxiv.org/abs/2603.10044","url_pdf":"https://arxiv.org/pdf/2603.10044","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2603.10044","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2603.10044"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/davidgringras/safety-under-scaffolding","reach":null}],"summary":{"unverified":6},"by_repo_kind":{"found_in_text":{"samples":6,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0afbd6dbecdf5dc6","entry":"cell_color","repo":"davidgringras/safety-under-scaffolding","repo_kind":"found_in_text","path":"paper_v2/figures/fig_safety_scorecard.py","file_url":"https://github.com/davidgringras/safety-under-scaffolding/blob/HEAD/paper_v2/figures/fig_safety_scorecard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0afbd6dbecdf5dc6"}},{"code_sha256_prefix":"b997ff8f38c7ccc0","entry":"compute_effect_sizes","repo":"davidgringras/safety-under-scaffolding","repo_kind":"found_in_text","path":"scaffold_safety/analysis/statistics.py","file_url":"https://github.com/davidgringras/safety-under-scaffolding/blob/HEAD/scaffold_safety/analysis/statistics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b997ff8f38c7ccc0"}},{"code_sha256_prefix":"f84cd7cdcc007aa5","entry":"compute_nnh","repo":"davidgringras/safety-under-scaffolding","repo_kind":"found_in_text","path":"scaffold_safety/analysis/statistics.py","file_url":"https://github.com/davidgringras/safety-under-scaffolding/blob/HEAD/scaffold_safety/analysis/statistics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f84cd7cdcc007aa5"}},{"code_sha256_prefix":"9d7727ffc6515243","entry":"compute_safety_rates","repo":"davidgringras/safety-under-scaffolding","repo_kind":"found_in_text","path":"scaffold_safety/analysis/statistics.py","file_url":"https://github.com/davidgringras/safety-under-scaffolding/blob/HEAD/scaffold_safety/analysis/statistics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"9d7727ffc6515243"}},{"code_sha256_prefix":"c03e496bae65251c","entry":"text_color","repo":"davidgringras/safety-under-scaffolding","repo_kind":"found_in_text","path":"paper_v2/figures/fig_safety_scorecard.py","file_url":"https://github.com/davidgringras/safety-under-scaffolding/blob/HEAD/paper_v2/figures/fig_safety_scorecard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c03e496bae65251c"}},{"code_sha256_prefix":"7f7200ee908b002b","entry":"wilson_ci_fallback","repo":"davidgringras/safety-under-scaffolding","repo_kind":"found_in_text","path":"paper_v2/figures/fig_safety_scorecard.py","file_url":"https://github.com/davidgringras/safety-under-scaffolding/blob/HEAD/paper_v2/figures/fig_safety_scorecard.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7f7200ee908b002b"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}