{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/value-residual-learning-for-alleviating","title":"Value Residual Learning For Alleviating Attention Concentration In Transformers","arxiv_id":"2410.17897","date":"2024-10-23","proceeding":null,"authors":["Zhanchao Zhou","Tianyi Wu","Zhiyun Jiang","Zhenzhong Lan"],"abstract":"Transformers can capture long-range dependencies using self-attention, allowing tokens to attend to all others directly. However, stacking multiple attention layers leads to attention concentration. One natural way to address this issue is to use cross-layer attention, allowing information from earlier layers to be directly accessible to later layers. However, this approach is computationally expensive. To address this problem, we propose Transformer with residual value (ResFormer) which approximates cross-layer attention through adding a residual connection from the values of the the first layer to all subsequent layers. Based on this method, one variant is the Transformer with single layer value (SVFormer), where all layers share the same value embedding from first layer. Comprehensive empirical evidence demonstrates ResFormer achieves equivalent validation loss with 10.4% fewer model parameters and 13.6% less training data compared to Transformer, while maintaining similar memory usage and computational cost. Besides, SVFormer reduces KV cache size by nearly half with only a small performance penalty and can be integrated with other KV-efficient methods, yielding further reductions in KV cache, with performance influenced by sequence length and cumulative learning rate. Further visualization results suggest that Resformer and SVFormer alleviate attention concentration in deeper layers through avoiding value-state drains and enhance representation across most layers.","url_abs":"https://arxiv.org/abs/2410.17897v3","url_pdf":"https://arxiv.org/pdf/2410.17897v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"value-residual-learning-for-alleviating","repo_url":"https://github.com/Zcchill/Value-Residual-Learning","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-sinks","method_name":"Attention Sinks"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.17897","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.17897"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/Zcchill/Value-Residual-Learning","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_fixture":1,"ran_draft_wrong":2,"unverified":9},"by_repo_kind":{"official":{"samples":12,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"30d7eec482ebf6b1","entry":"repeat_kv","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"src/modeling/modeling_llama_NeuTRENO_lambda04.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/src/modeling/modeling_llama_NeuTRENO_lambda04.py","link_basis":"harvester_set","language":"python","status":"ran_fixture","verification_level":2,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"30d7eec482ebf6b1"}},{"code_sha256_prefix":"bac65c3dafaec040","entry":"apply_rotary_pos_emb","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"src/modeling/modeling_llama_NeuTRENO_lambda04.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/src/modeling/modeling_llama_NeuTRENO_lambda04.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bac65c3dafaec040"}},{"code_sha256_prefix":"b99eea6376d1e212","entry":"rotate_half","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"src/modeling/modeling_llama_NeuTRENO_lambda04.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/src/modeling/modeling_llama_NeuTRENO_lambda04.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b99eea6376d1e212"}},{"code_sha256_prefix":"ed28cdaf289e0435","entry":"calculate_entropy","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"analyze/get_entropy.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/analyze/get_entropy.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ed28cdaf289e0435"}},{"code_sha256_prefix":"b486d0595afc6517","entry":"calculate_similarity","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"analyze/get_similarity.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/analyze/get_similarity.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b486d0595afc6517"}},{"code_sha256_prefix":"394fe2301da65126","entry":"data_generator_interleave","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"src/dataloaders.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/src/dataloaders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"394fe2301da65126"}},{"code_sha256_prefix":"f63c6075be4f4b9e","entry":"domain_data_gen","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"src/dataloaders.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/src/dataloaders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f63c6075be4f4b9e"}},{"code_sha256_prefix":"7beae4353ada6744","entry":"extract_loss","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"analyze/plot_relative_loss.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/analyze/plot_relative_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7beae4353ada6744"}},{"code_sha256_prefix":"5cf3265bba37e71f","entry":"get_perdomain_datasets","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"src/dataloaders.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/src/dataloaders.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5cf3265bba37e71f"}},{"code_sha256_prefix":"3c38d2d597def42a","entry":"get_scheduler_extended","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"src/trainer.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/src/trainer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3c38d2d597def42a"}},{"code_sha256_prefix":"eae14ebac1cd9c52","entry":"load_tf_weights_in_gpt2","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"src/modeling/modeling_gpt2_baseline.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/src/modeling/modeling_gpt2_baseline.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"eae14ebac1cd9c52"}},{"code_sha256_prefix":"01bbee0c31a1d848","entry":"moving_average","repo":"Zcchill/Value-Residual-Learning","repo_kind":"official","path":"analyze/plot_relative_loss.py","file_url":"https://github.com/Zcchill/Value-Residual-Learning/blob/HEAD/analyze/plot_relative_loss.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"01bbee0c31a1d848"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}