{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2606-26560","title":"Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention","arxiv_id":"2606.26560","date":"2026-06-25","proceeding":null,"authors":["Xiao Li","Chengruidong Zhang","Hao Luo","Xi Lin","Zekun Wang","Zihan Qiu","Yunfei Mao","Langshi Chen","Man Yuan","Minmin Sun","Huiqiang Jiang","Siqi Zhang","Rui Men","Wei Hu","Gong Cheng","Bo Zheng","Dayiheng Liu","Jingren Zhou"],"abstract":"Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active correction is still anchored to that same write address. As a result, stale information stored at a different address cannot be actively removed before new content is written elsewhere. We propose Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write. The key insight is that recurrent memory models should not only correct the current write, but also selectively suppress outdated memory at an independently chosen address. Concretely, our method first applies a targeted erase step along a learned erase direction, and then performs the standard delta-style corrective write along the current write direction. This preserves the corrective behavior of delta-rule updates while expanding their memory-management capacity. Language-model pretraining experiments across dense 2.5B and MoE 25B-A2.8B model families show that EDA performs best in both settings. The gain persists after 80B-token long-context midtraining of the MoE models, where EDA also performs best in long-context evaluations from 4k to 128k contexts. A compact update analysis and memory-state probes suggest why: EDA keeps the delta-rule corrective write intact while allocating an additional cleanup path most strongly when passive decay is weak. These results suggest that recurrent memory models should decide not only what to write, but also what stale information to erase and where.","url_abs":"https://arxiv.org/abs/2606.26560","url_pdf":"https://arxiv.org/pdf/2606.26560","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2606.26560","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2606.26560"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/QwenLM/FlashQLA","reach":null}],"summary":{"ran":6,"unverified":4},"by_repo_kind":{"found_in_text":{"samples":10,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6be3afdb318823ce","entry":"generate_rand_seqlens","repo":"QwenLM/FlashQLA","repo_kind":"found_in_text","path":"benchmark/bench_gated_delta_rule.py","file_url":"https://github.com/QwenLM/FlashQLA/blob/HEAD/benchmark/bench_gated_delta_rule.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6be3afdb318823ce"}},{"code_sha256_prefix":"1dd2c94ebadfb74b","entry":"input_guard","repo":"QwenLM/FlashQLA","repo_kind":"found_in_text","path":"flash_qla/utils/contiguous.py","file_url":"https://github.com/QwenLM/FlashQLA/blob/HEAD/flash_qla/utils/contiguous.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"1dd2c94ebadfb74b"}},{"code_sha256_prefix":"28c1d6470ca9d60e","entry":"pack","repo":"QwenLM/FlashQLA","repo_kind":"found_in_text","path":"flash_qla/utils/pack.py","file_url":"https://github.com/QwenLM/FlashQLA/blob/HEAD/flash_qla/utils/pack.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"28c1d6470ca9d60e"}},{"code_sha256_prefix":"6504d2035c704312","entry":"pad_and_reshape","repo":"QwenLM/FlashQLA","repo_kind":"found_in_text","path":"flash_qla/utils/pack.py","file_url":"https://github.com/QwenLM/FlashQLA/blob/HEAD/flash_qla/utils/pack.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6504d2035c704312"}},{"code_sha256_prefix":"94e6a53181d80161","entry":"tensor_cache","repo":"QwenLM/FlashQLA","repo_kind":"found_in_text","path":"flash_qla/utils/index.py","file_url":"https://github.com/QwenLM/FlashQLA/blob/HEAD/flash_qla/utils/index.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"94e6a53181d80161"}},{"code_sha256_prefix":"e1e90a347a0962b9","entry":"unpack","repo":"QwenLM/FlashQLA","repo_kind":"found_in_text","path":"flash_qla/utils/pack.py","file_url":"https://github.com/QwenLM/FlashQLA/blob/HEAD/flash_qla/utils/pack.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e1e90a347a0962b9"}},{"code_sha256_prefix":"f632515fd39572bf","entry":"l2norm_bwd_compiled","repo":"QwenLM/FlashQLA","repo_kind":"found_in_text","path":"flash_qla/utils/math.py","file_url":"https://github.com/QwenLM/FlashQLA/blob/HEAD/flash_qla/utils/math.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f632515fd39572bf"}},{"code_sha256_prefix":"7b4133146262db02","entry":"l2norm_fwd","repo":"QwenLM/FlashQLA","repo_kind":"found_in_text","path":"flash_qla/utils/math.py","file_url":"https://github.com/QwenLM/FlashQLA/blob/HEAD/flash_qla/utils/math.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7b4133146262db02"}},{"code_sha256_prefix":"d90be38c541e143f","entry":"l2norm_fwd_compiled","repo":"QwenLM/FlashQLA","repo_kind":"found_in_text","path":"flash_qla/utils/math.py","file_url":"https://github.com/QwenLM/FlashQLA/blob/HEAD/flash_qla/utils/math.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d90be38c541e143f"}},{"code_sha256_prefix":"02015534e010c30f","entry":"prepare_lens","repo":"QwenLM/FlashQLA","repo_kind":"found_in_text","path":"flash_qla/utils/index.py","file_url":"https://github.com/QwenLM/FlashQLA/blob/HEAD/flash_qla/utils/index.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"02015534e010c30f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.CL","source":"arxiv_2026.jsonl"},"syntology_extracted_results":{"kind":"leaderboard_placements","source":"Syntology's leaderboard-shaped extractor over the paper's own arXiv-HTML tables: a model pointed at a cell, the number was read from that cell and checked against the board's metric, dataset, split and scale, and an independent check accepted the entry; not reviewed by the paper's authors or the archive's editors","extractor_model":"global.anthropic.claude-sonnet-4-5-20250929-v1:0","verifier_model":null,"prompt_sha":"fa63d4bb9d755694","coverage":{"sentence":"Syntology has checked 6,264 of the 9,581 papers on this site that are newer than the archive; results from the others appear after they are checked.","papers_newer_than_archive":9581,"papers_checked":6264},"entries":[],"not_placed":{"boards":0,"rejected_by_independent_check":0,"refused_by_a_rule":0,"check_did_not_answer":0,"proposed_without_a_cell":0,"declined_by_site":0}}}