{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/dape-v2-process-attention-score-as-feature","title":"DAPE V2: Process Attention Score as Feature Map for Length Extrapolation","arxiv_id":"2410.04798","date":"2024-10-07","proceeding":null,"authors":["Chuanyang Zheng","Yihang Gao","Han Shi","Jing Xiong","Jiankai Sun","Jingyao Li","Minbin Huang","Xiaozhe Ren","Michael Ng","Xin Jiang","Zhenguo Li","Yu Li"],"abstract":"The attention mechanism is a fundamental component of the Transformer model, contributing to interactions among distinct tokens, in contrast to earlier feed-forward neural networks. In general, the attention scores are determined simply by the key-query products. However, this work's occasional trial (combining DAPE and NoPE) of including additional MLPs on attention scores without position encoding indicates that the classical key-query multiplication may limit the performance of Transformers. In this work, we conceptualize attention as a feature map and apply the convolution operator (for neighboring attention scores across different heads) to mimic the processing methods in computer vision. Specifically, the main contribution of this paper is identifying and interpreting the Transformer length extrapolation problem as a result of the limited expressiveness of the naive query and key dot product, and we successfully translate the length extrapolation issue into a well-understood feature map processing problem. The novel insight, which can be adapted to various attention-related models, reveals that the current Transformer architecture has the potential for further evolution. Extensive experiments demonstrate that treating attention as a feature map and applying convolution as a processing method significantly enhances Transformer performance.","url_abs":"https://arxiv.org/abs/2410.04798v3","url_pdf":"https://arxiv.org/pdf/2410.04798v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"dape-v2-process-attention-score-as-feature","repo_url":"https://github.com/chuanyang-zheng/dape","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"dape-v2-process-attention-score-as-feature","repo_url":"https://github.com/chuanyang-zheng/cape","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"convolution","method_name":"Convolution"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.04798","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.04798"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chuanyang-zheng/cape","reach":{"status":"ok","spdx":"Apache-2.0"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/chuanyang-zheng/dape","reach":null}],"summary":{"ran":4,"unverified":6},"by_repo_kind":{"listed":{"samples":10,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"eb7511606bf2a37e","entry":"bias_gelu","repo":"chuanyang-zheng/cape","repo_kind":"listed","path":"megatron/model/activations.py","file_url":"https://github.com/chuanyang-zheng/cape/blob/HEAD/megatron/model/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"eb7511606bf2a37e"}},{"code_sha256_prefix":"5dc15a21043ef7e3","entry":"bias_gelu_back","repo":"chuanyang-zheng/cape","repo_kind":"listed","path":"megatron/model/activations.py","file_url":"https://github.com/chuanyang-zheng/cape/blob/HEAD/megatron/model/activations.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"5dc15a21043ef7e3"}},{"code_sha256_prefix":"6464b20466536553","entry":"get_flops","repo":"chuanyang-zheng/cape","repo_kind":"listed","path":"megatron/logging.py","file_url":"https://github.com/chuanyang-zheng/cape/blob/HEAD/megatron/logging.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"6464b20466536553"}},{"code_sha256_prefix":"32c8c6d61ee207b7","entry":"human_readable_flops","repo":"chuanyang-zheng/cape","repo_kind":"listed","path":"megatron/logging.py","file_url":"https://github.com/chuanyang-zheng/cape/blob/HEAD/megatron/logging.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"32c8c6d61ee207b7"}},{"code_sha256_prefix":"9a2bd4d6d6d05bf6","entry":"get_activation","repo":"chuanyang-zheng/cape","repo_kind":"listed","path":"megatron/model/activations.py","file_url":"https://github.com/chuanyang-zheng/cape/blob/HEAD/megatron/model/activations.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"9a2bd4d6d6d05bf6"}},{"code_sha256_prefix":"69002400aad5a8a6","entry":"get_checkpoint_tag","repo":"chuanyang-zheng/cape","repo_kind":"listed","path":"megatron/checkpointing.py","file_url":"https://github.com/chuanyang-zheng/cape/blob/HEAD/megatron/checkpointing.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"69002400aad5a8a6"}},{"code_sha256_prefix":"d5081f28e2befd2c","entry":"get_docs","repo":"chuanyang-zheng/cape","repo_kind":"listed","path":"configs/gen_docs.py","file_url":"https://github.com/chuanyang-zheng/cape/blob/HEAD/configs/gen_docs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d5081f28e2befd2c"}},{"code_sha256_prefix":"3a5fc202b5d9ec24","entry":"monitor_method_cuda_wall_times","repo":"chuanyang-zheng/cape","repo_kind":"listed","path":"megatron/devutil.py","file_url":"https://github.com/chuanyang-zheng/cape/blob/HEAD/megatron/devutil.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"3a5fc202b5d9ec24"}},{"code_sha256_prefix":"50c9c165cf801f59","entry":"pairwise","repo":"chuanyang-zheng/cape","repo_kind":"listed","path":"configs/gen_docs.py","file_url":"https://github.com/chuanyang-zheng/cape/blob/HEAD/configs/gen_docs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"50c9c165cf801f59"}},{"code_sha256_prefix":"64dd872a8ac47294","entry":"to_md","repo":"chuanyang-zheng/cape","repo_kind":"listed","path":"configs/gen_docs.py","file_url":"https://github.com/chuanyang-zheng/cape/blob/HEAD/configs/gen_docs.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"64dd872a8ac47294"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}