{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-11063","title":"CIAware-Bench: Benchmarking Control Intervention Awareness Across Frontier LLMs","arxiv_id":"2606.11063","date":"2026-06-09","proceeding":null,"authors":["Joachim Schaeffer","Thomas Jiralerspong","Alexander Panfilov","Guillaume Lajoie","Jonas Geiping","Yoshua Bengio","Roland S. Zimmermann"],"abstract":"AI control protocols oversee untrusted models by monitoring their actions and modifying potentially unsafe steps, often using a trusted model. This partially tampers with the untrusted model's trajectory. If the trusted model detects such an intervention, it may infer properties of the monitor and adapt to evade control. We introduce \\textbf{CIAware-Bench}, a benchmark for measuring \\textbf{c}ontrol \\textbf{i}ntervention (CI) awareness across frontier models. CIAware-Bench tests whether models can distinguish their own trajectories from those modified by a control intervention. The benchmark is comprised of a suite of four task domains (essay writing, BigCodeBench, Bash Arena, and SHADE-Arena), while varying trajectory watermarking, side-task presence, and the control protocol. Evaluating eleven frontier models, we find low to moderate CI awareness under default settings (up to 0.87; random chance balanced binary classification accuracy is 0.5) with substantial variation across task domains and model pairs. Detection is generally easier across model families, suggesting that models exploit provider-specific differences in style or post-training. Overall, CI awareness is not a fixed model-level property, and should be measured for each new model release and deployment scenario. We release CIAware-Bench to track CI awareness and inform control protocols whose interventions are harder to detect.","url_abs":"https://arxiv.org/abs/2606.11063","url_pdf":"https://arxiv.org/pdf/2606.11063","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2606.11063","atlas_url":"https://app.syntology.ai/?focus=2606.11063","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.11063"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/JoachimSchaeffer/CIAware-Bench","reach":null}],"summary":{"ran":16,"unverified":8},"by_repo_kind":{"found_in_text":{"samples":24,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":24,"samples":[{"code_sha256_prefix":"4c37cd0e3635728c","entry":"bcb_model_slug","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/paths.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/paths.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4c37cd0e3635728c"}},{"code_sha256_prefix":"7faa7e98c2376c76","entry":"binomial_test_pvalue","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/analysis/statistics.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/analysis/statistics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7faa7e98c2376c76"}},{"code_sha256_prefix":"9cf73bc7fe2412b5","entry":"cohens_h","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/analysis/statistics.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/analysis/statistics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9cf73bc7fe2412b5"}},{"code_sha256_prefix":"0aa1c47293980174","entry":"compute_auroc","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/analysis/metrics.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/analysis/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0aa1c47293980174"}},{"code_sha256_prefix":"ad0b36e47e8b4ff1","entry":"format_msg","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/analysis/compare_interventions.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/analysis/compare_interventions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ad0b36e47e8b4ff1"}},{"code_sha256_prefix":"fe9897f7e960225f","entry":"format_step","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/analysis/compare_interventions.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/analysis/compare_interventions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fe9897f7e960225f"}},{"code_sha256_prefix":"2463d7873bd2cb17","entry":"format_tool_result","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/analysis/compare_interventions.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/analysis/compare_interventions.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2463d7873bd2cb17"}},{"code_sha256_prefix":"443bcea492142e4b","entry":"import_control_arena_module","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/control_arena_compat.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/control_arena_compat.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"443bcea492142e4b"}},{"code_sha256_prefix":"13a56e8066c8a591","entry":"model_short_name","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/paths.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/paths.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"13a56e8066c8a591"}},{"code_sha256_prefix":"601ae1fe357d2b32","entry":"parse_config_path","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/config.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"601ae1fe357d2b32"}},{"code_sha256_prefix":"7e91e53b7bf85841","entry":"parse_elicitation_level","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/types.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/types.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7e91e53b7bf85841"}},{"code_sha256_prefix":"fabab3fafc6e54b6","entry":"parse_intervention_type","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/types.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/types.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"fabab3fafc6e54b6"}},{"code_sha256_prefix":"212eeff1b4f87493","entry":"path_slug","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/paths.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/paths.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"212eeff1b4f87493"}},{"code_sha256_prefix":"476c24d0a1a25765","entry":"resolve_intervention_max","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/config.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/config.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"476c24d0a1a25765"}},{"code_sha256_prefix":"17c5fcf026829405","entry":"summarize_probe_samples","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/analysis/metrics.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/analysis/metrics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"17c5fcf026829405"}},{"code_sha256_prefix":"be4cb87871c7ea43","entry":"wilson_confidence_interval","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/analysis/statistics.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/analysis/statistics.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"be4cb87871c7ea43"}},{"code_sha256_prefix":"b74c8d8c3a70128d","entry":"compute_accuracy_by_group","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/analysis/metrics.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/analysis/metrics.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b74c8d8c3a70128d"}},{"code_sha256_prefix":"828842fb935b2ab7","entry":"environment_names_with","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/environments.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/environments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"828842fb935b2ab7"}},{"code_sha256_prefix":"cac6395112980cd5","entry":"environment_supports","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/environments.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/environments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"cac6395112980cd5"}},{"code_sha256_prefix":"846d6e9e7be62cfd","entry":"get_environment_spec","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/environments.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/environments.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"846d6e9e7be62cfd"}},{"code_sha256_prefix":"b27d5b8765596fce","entry":"get_model","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/model_registry.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/model_registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b27d5b8765596fce"}},{"code_sha256_prefix":"14de90c23c9d2544","entry":"load_config","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/config.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"14de90c23c9d2544"}},{"code_sha256_prefix":"96aec4f897586502","entry":"plot_label","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/model_registry.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/model_registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"96aec4f897586502"}},{"code_sha256_prefix":"f69f7c6949575740","entry":"plot_sort_key","repo":"JoachimSchaeffer/CIAware-Bench","repo_kind":"found_in_text","path":"cia/core/model_registry.py","file_url":"https://github.com/JoachimSchaeffer/CIAware-Bench/blob/HEAD/cia/core/model_registry.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f69f7c6949575740"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}