{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2608-31017","title":"One note in three: a verified census of three deployed AI scribes, and the instrument that counted it","arxiv_id":"2608.31017","date":"2026-08-31","proceeding":null,"authors":["Sebastian Fox","Luke Markham","Ryan Lail","Michael Karotsieris"],"abstract":"Ambient AI scribes draft clinical notes under the reassurance that a clinician signs every note. We audited three commercial AI scribes on the same 142 consultations: 565 notes from recorded UK primary-care and US ambulatory encounters plus authored scenarios. Twelve discovery passes proposed 13,678 candidate errors; the 5,898 clearing an importance filter went to an adversarial panel of two models from different families, each told to refute what it could, and 618 survived. One note in three (31.3% [27.0, 35.6]) carries a verified failure, concentrated in allergy and medication information, invented patient identity, and history written up as examination on telephone consultations that can contain none. No product was given a patient record; setting aside the two classes a record would have prefilled, invented identity and dates, the rate is 24.8% [20.8, 29.0]. One failure mode did not fit our scheme, drawn from published scribe-error taxonomies: a treatment the clinician retracts, recorded as delivered care. Two clinicians adjudicated blind, disjoint samples: a physician author upheld 20 of 21 findings (95.2% [77.3, 99.2]) and an independent clinician, not an author, 12 of 12 ([75.8, 100]); both judged every sampled refusal genuine. A failure rate depends on the instrument as much as the scribes. With model, evidence and settings fixed, the review instruction alone moves the share of candidates verified from 9.3% to 79.0%, and the reviewing family moves it too: alone at that instruction the gentler flags 54.8% of notes against 27.8%. Between 28% and 97% of sampled notes carry a failure depending on the standard. Published audits disagree among themselves by a margin instrument differences alone can produce: omission is 54-86% of their errors against our 23.1%. We release all 618 findings with transcript-side evidence, every prompt and model version, and the re-runnable pipeline.","url_abs":"https://arxiv.org/abs/2608.31017","url_pdf":"https://arxiv.org/pdf/2608.31017","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2608.31017","atlas_url":"https://app.syntology.ai/?focus=2608.31017","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2608.31017"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/composo-ai/omission-bench","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":21},"by_repo_kind":{"found_in_text":{"samples":21,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"00705670768d17c3","entry":"alnum","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_medec_ingest.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_medec_ingest.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"00705670768d17c3"}},{"code_sha256_prefix":"20a42bb3322e5d79","entry":"claude","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"common.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20a42bb3322e5d79"}},{"code_sha256_prefix":"e24fef41115ce0da","entry":"claude_json","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"common.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e24fef41115ce0da"}},{"code_sha256_prefix":"ab13f31dde7d0540","entry":"fact_uid","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"benchmark/omission_factorial.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/benchmark/omission_factorial.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ab13f31dde7d0540"}},{"code_sha256_prefix":"0003639f9244a653","entry":"flagged_conservative","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_medec_analyze.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_medec_analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0003639f9244a653"}},{"code_sha256_prefix":"bc07923772d57ff0","entry":"has_annotation","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_medval_ingest.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_medval_ingest.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc07923772d57ff0"}},{"code_sha256_prefix":"ec19f94e70863108","entry":"load_primock","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"common.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ec19f94e70863108"}},{"code_sha256_prefix":"727005434407d893","entry":"medec_flag","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_adapters.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_adapters.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"727005434407d893"}},{"code_sha256_prefix":"86185f382a8abc68","entry":"medval_any_task","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_adapters.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_adapters.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"86185f382a8abc68"}},{"code_sha256_prefix":"4a94a5ef6ee42386","entry":"medval_d2n","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_adapters.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_adapters.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"4a94a5ef6ee42386"}},{"code_sha256_prefix":"fab001a6ddc36352","entry":"norm","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"benchmark/omission_sites.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/benchmark/omission_sites.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fab001a6ddc36352"}},{"code_sha256_prefix":"d849b970da50845a","entry":"norm_assessment","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_medval_ingest.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_medval_ingest.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d849b970da50845a"}},{"code_sha256_prefix":"8b4149dc9f853083","entry":"normlow","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"benchmark/omission_sites.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/benchmark/omission_sites.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8b4149dc9f853083"}},{"code_sha256_prefix":"46d6554caeb8975f","entry":"parse_cell","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"w2_common.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/w2_common.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46d6554caeb8975f"}},{"code_sha256_prefix":"3516fd5d0dc889c6","entry":"parse_sentences","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_medec_ingest.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_medec_ingest.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3516fd5d0dc889c6"}},{"code_sha256_prefix":"7ed3f5cae5e5b79d","entry":"residual_options","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"benchmark/omission_factorial.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/benchmark/omission_factorial.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ed3f5cae5e5b79d"}},{"code_sha256_prefix":"c55027e2e9dbd641","entry":"roc","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_medec_analyze.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_medec_analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c55027e2e9dbd641"}},{"code_sha256_prefix":"7ae530b94f0d11ea","entry":"sha256","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_medval_ingest.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_medval_ingest.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7ae530b94f0d11ea"}},{"code_sha256_prefix":"2af28ebd913fc258","entry":"stratified","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"census/taxonomy_omission_audit.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/census/taxonomy_omission_audit.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2af28ebd913fc258"}},{"code_sha256_prefix":"2e531ca1cb2f2ff4","entry":"two_prop_z","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_medec_analyze.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_medec_analyze.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"2e531ca1cb2f2ff4"}},{"code_sha256_prefix":"91342c80287e9db5","entry":"ws","repo":"composo-ai/omission-bench","repo_kind":"found_in_text","path":"anchors/wb_medec_ingest.py","file_url":"https://github.com/composo-ai/omission-bench/blob/HEAD/anchors/wb_medec_ingest.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"91342c80287e9db5"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_daily_20260901.json"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,885 of the 9,623 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9623,"papers_checked":6885},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}