{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/towards-better-multi-head-attention-via","title":"Towards Better Multi-head Attention via Channel-wise Sample Permutation","arxiv_id":"2410.10914","date":"2024-10-14","proceeding":null,"authors":["Shen Yuan","Hongteng Xu"],"abstract":"Transformer plays a central role in many fundamental deep learning models, e.g., the ViT in computer vision and the BERT and GPT in natural language processing, whose effectiveness is mainly attributed to its multi-head attention (MHA) mechanism. In this study, we propose a simple and novel channel-wise sample permutation (CSP) operator, achieving a new structured MHA with fewer parameters and lower complexity. Given an input matrix, CSP circularly shifts the samples of different channels with various steps and then sorts grouped samples of each channel. This operator is equivalent to implicitly implementing cross-channel attention maps as permutation matrices, which achieves linear complexity and suppresses the risk of rank collapse when representing data. We replace the MHA of some representative models with CSP and test the CSP-based models in several discriminative tasks, including image classification and long sequence analysis. Experiments show that the CSP-based models achieve comparable or better performance with fewer parameters and lower computational costs than the classic Transformer and its state-of-the-art variants. The code is available at https://github.com/DaShenZi721/CSP.","url_abs":"https://arxiv.org/abs/2410.10914v1","url_pdf":"https://arxiv.org/pdf/2410.10914v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"towards-better-multi-head-attention-via","repo_url":"https://github.com/dashenzi721/csp","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bert","method_name":"BERT"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"discriminative-fine-tuning","method_name":"Discriminative Fine-Tuning"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt","method_name":"GPT"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"linear-warmup-with-linear-decay","method_name":"Linear Warmup With Linear Decay"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"weight-decay","method_name":"Weight Decay"},{"method_slug":"wordpiece","method_name":"WordPiece"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2410.10914","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2410.10914"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/dashenzi721/csp","reach":{"status":"ok"}}],"summary":{"ran":4,"unverified":3},"by_repo_kind":{"official":{"samples":7,"ran":4,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"07969f7041022050","entry":"distributed_init","repo":"dashenzi721/csp","repo_kind":"official","path":"lra/mega_csp/fairseq/distributed_utils.py","file_url":"https://github.com/dashenzi721/csp/blob/HEAD/lra/mega_csp/fairseq/distributed_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"07969f7041022050"}},{"code_sha256_prefix":"90b79fe15491de83","entry":"is_master","repo":"dashenzi721/csp","repo_kind":"official","path":"lra/mega_csp/fairseq/distributed_utils.py","file_url":"https://github.com/dashenzi721/csp/blob/HEAD/lra/mega_csp/fairseq/distributed_utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"90b79fe15491de83"}},{"code_sha256_prefix":"82bdf940665aeeb4","entry":"safe_readline","repo":"dashenzi721/csp","repo_kind":"official","path":"lra/mega_csp/fairseq/binarizer.py","file_url":"https://github.com/dashenzi721/csp/blob/HEAD/lra/mega_csp/fairseq/binarizer.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"82bdf940665aeeb4"}},{"code_sha256_prefix":"9418b43048e4e4ec","entry":"url_to_filename","repo":"dashenzi721/csp","repo_kind":"official","path":"lra/mega_csp/fairseq/file_utils.py","file_url":"https://github.com/dashenzi721/csp/blob/HEAD/lra/mega_csp/fairseq/file_utils.py","link_basis":"plan_row","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9418b43048e4e4ec"}},{"code_sha256_prefix":"199e96db5edd0edd","entry":"all_reduce","repo":"dashenzi721/csp","repo_kind":"official","path":"lra/mega_csp/fairseq/distributed_utils.py","file_url":"https://github.com/dashenzi721/csp/blob/HEAD/lra/mega_csp/fairseq/distributed_utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"199e96db5edd0edd"}},{"code_sha256_prefix":"b1a033619f614e17","entry":"filename_to_url","repo":"dashenzi721/csp","repo_kind":"official","path":"lra/mega_csp/fairseq/file_utils.py","file_url":"https://github.com/dashenzi721/csp/blob/HEAD/lra/mega_csp/fairseq/file_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b1a033619f614e17"}},{"code_sha256_prefix":"bd3fe80662b28bc7","entry":"load_archive_file","repo":"dashenzi721/csp","repo_kind":"official","path":"lra/mega_csp/fairseq/file_utils.py","file_url":"https://github.com/dashenzi721/csp/blob/HEAD/lra/mega_csp/fairseq/file_utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bd3fe80662b28bc7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}