{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-02083","title":"EditPropBench: Measuring Factual Edit Propagation in Scientific Manuscripts","arxiv_id":"2605.02083","date":"2026-05-03","proceeding":null,"authors":["Garvin Kruthof"],"abstract":"Local factual edits in scientific manuscripts often create non-local revision obligations. If a dataset changes from 215 to 80 documents, claims such as 'medium-scale' or 'a few hundred items' may also become stale, even though they do not repeat the edited number. In an audit of recent arXiv cs.CL benchmark and dataset papers, we find fact-dependent qualitative claims in 37.2% of papers, suggesting that this dependency pattern is common in the target genre. We introduce EditPropBench, a benchmark for measuring whether LLM editors propagate factual edits through dependent manuscript claims. Each item contains an ML/NLP-style synthetic manuscript, a targeted edit, and a controlled fact graph with sentence-level labels for direct targets, required downstream updates, and unrelated text that should remain unchanged. We summarize cascade success with Edit-Ripple Adherence (ERA), the fraction of required downstream updates correctly revised, and validate the metric with adversarial probes and stress-test variants. On the hardest cases, where dependent claims use implicit or free-form wording rather than repeating the edited value, five LLM editing systems span ERA 0.148-0.705. Even the strongest misses roughly 30% of required cascade updates. This advantage persists in a mixed evaluation that includes easy cases solvable by deterministic substitution. EditPropBench shows that current LLM editors can repair many implicit consequences of factual edits, but reliable scientific revision still requires cascade-aware checking.","url_abs":"https://arxiv.org/abs/2605.02083","url_pdf":"https://arxiv.org/pdf/2605.02083","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.02083","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.02083"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/kruthof/EditCoherenceBench","reach":null}],"summary":{"ran":9,"ran_draft_wrong":4,"ran_violates":1,"unverified":14},"by_repo_kind":{"found_in_text":{"samples":28,"ran":14,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":28,"samples":[{"code_sha256_prefix":"37094dc8345ed6c2","entry":"Claim","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"37094dc8345ed6c2"}},{"code_sha256_prefix":"e1f6d2bcd254ad8f","entry":"DependencyEdge","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e1f6d2bcd254ad8f"}},{"code_sha256_prefix":"dfca3ef1360e156e","entry":"DocumentUnit","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"dfca3ef1360e156e"}},{"code_sha256_prefix":"d8e6c2e3ddf1e3a7","entry":"Fact","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d8e6c2e3ddf1e3a7"}},{"code_sha256_prefix":"d20494e0a07c59dc","entry":"FigureSpec","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d20494e0a07c59dc"}},{"code_sha256_prefix":"7a3de1d2e21ad9a9","entry":"Frozen","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"7a3de1d2e21ad9a9"}},{"code_sha256_prefix":"9ed0fdb3323d626d","entry":"ManuscriptMetadata","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9ed0fdb3323d626d"}},{"code_sha256_prefix":"6476544361228475","entry":"NewUnit","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6476544361228475"}},{"code_sha256_prefix":"e69cd5eff6b21e67","entry":"TableSpec","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"e69cd5eff6b21e67"}},{"code_sha256_prefix":"51da8e7b162a21f0","entry":"_difficulty_for","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"51da8e7b162a21f0"}},{"code_sha256_prefix":"3c161d0359b0b840","entry":"_supports_family","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3c161d0359b0b840"}},{"code_sha256_prefix":"4335beea0eb46870","entry":"derived_model_pool_for","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"4335beea0eb46870"}},{"code_sha256_prefix":"8be514bf3ca41ce2","entry":"derived_qualifier_for","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8be514bf3ca41ce2"}},{"code_sha256_prefix":"31e8330ecef2f9b1","entry":"derived_qualifier_for_fact","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"31e8330ecef2f9b1"}},{"code_sha256_prefix":"9a9b924bc6de9b85","entry":"ComplexityTier","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9a9b924bc6de9b85"}},{"code_sha256_prefix":"0658b06d7db4f1cd","entry":"Difficulty","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0658b06d7db4f1cd"}},{"code_sha256_prefix":"03d0977d3a070323","entry":"EditFamily","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"03d0977d3a070323"}},{"code_sha256_prefix":"35e055fa2c2e7b08","entry":"EditItem","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"35e055fa2c2e7b08"}},{"code_sha256_prefix":"5d9fec9afdd3b417","entry":"FactType","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5d9fec9afdd3b417"}},{"code_sha256_prefix":"3e3d6e79bd0bdd60","entry":"LengthTier","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3e3d6e79bd0bdd60"}},{"code_sha256_prefix":"f7bec01a3c1b6aed","entry":"Manuscript","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f7bec01a3c1b6aed"}},{"code_sha256_prefix":"20da1cb98e985fe4","entry":"UnitType","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"20da1cb98e985fe4"}},{"code_sha256_prefix":"71aa1cbf4fb9358e","entry":"_instruction_for","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"71aa1cbf4fb9358e"}},{"code_sha256_prefix":"85b2cb21b5d90c66","entry":"_make_edit_item","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"85b2cb21b5d90c66"}},{"code_sha256_prefix":"99c69f487be4faa1","entry":"_new_value_for","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99c69f487be4faa1"}},{"code_sha256_prefix":"abf4c1bcfd8bdea3","entry":"derive_seed","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"abf4c1bcfd8bdea3"}},{"code_sha256_prefix":"a8463ac1f544a361","entry":"sample_edits","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8463ac1f544a361"}},{"code_sha256_prefix":"3b498419563f0c04","entry":"seeded_rng","repo":"kruthof/EditCoherenceBench","repo_kind":"found_in_text","path":"editpropbench/generator/edit_sampler.py","file_url":"https://github.com/kruthof/EditCoherenceBench/blob/HEAD/editpropbench/generator/edit_sampler.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"3b498419563f0c04"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}