{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-18855","title":"Delta Attention Residuals","arxiv_id":"2605.18855","date":"2026-05-13","proceeding":null,"authors":["Cheng Luo","Zefan Cai","Junjie Hu"],"abstract":"Attention Residuals replace standard additive residual connections with learned softmax attention over previous layer outputs, enabling selective cross-layer routing. However, standard Attention Residuals still attend over cumulative hidden states in previous layers, which are highly redundant. We show that this redundancy leads to routing collapse in deeper layers: attention weights become low-contrast and closer to uniform (max weight ${\\approx}$0.2), limiting the model's ability to select informative states in previous layers. This raises a key but underexplored design question: what layer-wise representations should be routed in Attention Residuals? To answer this question, we propose Delta Attention Residuals, which attend over deltas -- the change introduced by each sublayer ($\\mathbf{v}_i = \\mathbf{h}_{i+1} - \\mathbf{h}_i$) -- instead of cumulative states. Delta representations are structurally diverse and yield higher-contrast attention distributions (max weight ${\\approx}$0.6), enabling more selective and effective routing across layers. This principle applies at both per-sublayer and block granularity. Across all tested scales (220M--7.6B), Delta Attention Residuals consistently outperform both standard residuals and Attention Residuals, with 1.7--8.2\\% validation perplexity gains. Delta Attention Residuals also enables converting pretrained checkpoints into Delta Attention Residuals via standard fine-tuning. Code is available at https://github.com/wdlctc/delta-attention-residuals-code.","url_abs":"https://arxiv.org/abs/2605.18855","url_pdf":"https://arxiv.org/pdf/2605.18855","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.18855","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.18855"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/wdlctc/delta-attention-residuals-code","reach":null}],"summary":{"ran_fixture":2,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":3,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":3,"samples":[{"code_sha256_prefix":"089fdfe024fdd0bf","entry":"_delta_attn_res_kernel","repo":"wdlctc/delta-attention-residuals-code","repo_kind":"found_in_text","path":"Attention-Residuals/modeling_qwen3_attnres.py","file_url":"https://github.com/wdlctc/delta-attention-residuals-code/blob/HEAD/Attention-Residuals/modeling_qwen3_attnres.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"089fdfe024fdd0bf"}},{"code_sha256_prefix":"32122d4ce086f761","entry":"_delta_attn_res_kernel_with_entropy","repo":"wdlctc/delta-attention-residuals-code","repo_kind":"found_in_text","path":"Attention-Residuals/modeling_qwen3_attnres.py","file_url":"https://github.com/wdlctc/delta-attention-residuals-code/blob/HEAD/Attention-Residuals/modeling_qwen3_attnres.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"32122d4ce086f761"}},{"code_sha256_prefix":"24e6ce1c84c494ea","entry":"delta_attn_res","repo":"wdlctc/delta-attention-residuals-code","repo_kind":"found_in_text","path":"Attention-Residuals/modeling_qwen3_attnres.py","file_url":"https://github.com/wdlctc/delta-attention-residuals-code/blob/HEAD/Attention-Residuals/modeling_qwen3_attnres.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"24e6ce1c84c494ea"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}