{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/accelerating-neural-transformer-via-an","title":"Accelerating Neural Transformer via an Average Attention Network","arxiv_id":"1805.00631","date":"2018-05-02","proceeding":"ACL 2018 7","authors":["Biao Zhang","Deyi Xiong","Jinsong Su"],"abstract":"With parallelizable attention networks, the neural Transformer is very fast\nto train. However, due to the auto-regressive architecture and self-attention\nin the decoder, the decoding procedure becomes slow. To alleviate this issue,\nwe propose an average attention network as an alternative to the self-attention\nnetwork in the decoder of the neural Transformer. The average attention network\nconsists of two layers, with an average layer that models dependencies on\nprevious positions and a gating layer that is stacked over the average layer to\nenhance the expressiveness of the proposed attention network. We apply this\nnetwork on the decoder part of the neural Transformer to replace the original\ntarget-side self-attention model. With masking tricks and dynamic programming,\nour model enables the neural Transformer to decode sentences over four times\nfaster than its original version with almost no loss in training time and\ntranslation performance. We conduct a series of experiments on WMT17\ntranslation tasks, where on 6 different language pairs, we obtain robust and\nconsistent speed-ups in decoding.","url_abs":"http://arxiv.org/abs/1805.00631v3","url_pdf":"http://arxiv.org/pdf/1805.00631v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"accelerating-neural-transformer-via-an","repo_url":"https://github.com/bzhangXMU/transformer-aan","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"tasks":[{"task_slug":"decoder","task_name":"Decoder"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"Average Attention Network","rank_in_archive_order":60,"of":91,"metrics":{"BLEU score":"26.31"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"Average Attention Network (w/o FFN)","rank_in_archive_order":63,"of":91,"metrics":{"BLEU score":"26.05"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"Average Attention Network (w/o gate)","rank_in_archive_order":65,"of":91,"metrics":{"BLEU score":"25.91"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1805.00631","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1805.00631"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/bzhangXMU/transformer-aan","reach":{"status":"ok","spdx":"BSD-3-Clause"}}],"summary":{"unverified":19},"by_repo_kind":{"official":{"samples":19,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d159075513daf35b","entry":"add_timing_signal","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/layers/attention.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/layers/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"d159075513daf35b"}},{"code_sha256_prefix":"dd985e59ae3a38a4","entry":"closest_length","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/utils/bleu.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/utils/bleu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"dd985e59ae3a38a4"}},{"code_sha256_prefix":"c55c6aba7b709f18","entry":"collect_params","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/trainer.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"c55c6aba7b709f18"}},{"code_sha256_prefix":"273d5e370a58f518","entry":"combine_heads","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/layers/attention.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/layers/attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"273d5e370a58f518"}},{"code_sha256_prefix":"6cc541726e9562c3","entry":"compute_batch_indices","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/utils/search_aan.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/utils/search_aan.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"6cc541726e9562c3"}},{"code_sha256_prefix":"90f2d69b85bf60d5","entry":"compute_topk_scores_and_seq","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/utils/search_aan.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/utils/search_aan.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"90f2d69b85bf60d5"}},{"code_sha256_prefix":"1b441b0d053c905a","entry":"get_weights","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/models/transformer.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/models/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"1b441b0d053c905a"}},{"code_sha256_prefix":"149a90364d6911d9","entry":"import_params","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/translator.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/translator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"149a90364d6911d9"}},{"code_sha256_prefix":"ba3612d14359b1ea","entry":"import_params","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/trainer.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"ba3612d14359b1ea"}},{"code_sha256_prefix":"788092fa1ba69066","entry":"layer_norm","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/layers/nn.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/layers/nn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"788092fa1ba69066"}},{"code_sha256_prefix":"9676af2224c507fe","entry":"linear","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/layers/nn.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/layers/nn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9676af2224c507fe"}},{"code_sha256_prefix":"b4a5e930784a8355","entry":"log_prob_from_logits","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/utils/search_aan.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/utils/search_aan.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"b4a5e930784a8355"}},{"code_sha256_prefix":"52b62c3783732f3f","entry":"maxout","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/layers/nn.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/layers/nn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"52b62c3783732f3f"}},{"code_sha256_prefix":"8be2667aba5e919e","entry":"merge_parameters","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/translator.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/translator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8be2667aba5e919e"}},{"code_sha256_prefix":"9c5e87513bca1bae","entry":"modified_precision","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/utils/bleu.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/utils/bleu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"9c5e87513bca1bae"}},{"code_sha256_prefix":"69dd5f64c584f7eb","entry":"parse_args","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/trainer.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/trainer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"69dd5f64c584f7eb"}},{"code_sha256_prefix":"f3ce51f764fb2afc","entry":"residual_fn","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/models/transformer.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/models/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"f3ce51f764fb2afc"}},{"code_sha256_prefix":"531aab5936f2e596","entry":"shortest_length","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/utils/bleu.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/utils/bleu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"531aab5936f2e596"}},{"code_sha256_prefix":"98b6d2c44a099591","entry":"split_heads","repo":"bzhangXMU/transformer-aan","repo_kind":"official","path":"code/thumt/layers/attention.py","file_url":"https://github.com/bzhangXMU/transformer-aan/blob/HEAD/code/thumt/layers/attention.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"98b6d2c44a099591"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}