{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/bi-directional-block-self-attention-for-fast","title":"Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling","arxiv_id":"1804.00857","date":"2018-04-03","proceeding":"ICLR 2018 1","authors":["Tao Shen","Tianyi Zhou","Guodong Long","Jing Jiang","Chengqi Zhang"],"abstract":"Recurrent neural networks (RNN), convolutional neural networks (CNN) and\nself-attention networks (SAN) are commonly used to produce context-aware\nrepresentations. RNN can capture long-range dependency but is hard to\nparallelize and not time-efficient. CNN focuses on local dependency but does\nnot perform well on some tasks. SAN can model both such dependencies via highly\nparallelizable computation, but memory requirement grows rapidly in line with\nsequence length. In this paper, we propose a model, called \"bi-directional\nblock self-attention network (Bi-BloSAN)\", for RNN/CNN-free sequence encoding.\nIt requires as little memory as RNN but with all the merits of SAN. Bi-BloSAN\nsplits the entire sequence into blocks, and applies an intra-block SAN to each\nblock for modeling local context, then applies an inter-block SAN to the\noutputs for all blocks to capture long-range dependency. Thus, each SAN only\nneeds to process a short sequence, and only a small amount of memory is\nrequired. Additionally, we use feature-level attention to handle the variation\nof contexts around the same word, and use forward/backward masks to encode\ntemporal order information. On nine benchmark datasets for different NLP tasks,\nBi-BloSAN achieves or improves upon state-of-the-art accuracy, and shows better\nefficiency-memory trade-off than existing RNN/CNN/SAN.","url_abs":"http://arxiv.org/abs/1804.00857v1","url_pdf":"http://arxiv.org/pdf/1804.00857v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"bi-directional-block-self-attention-for-fast","repo_url":"https://github.com/taoshen58/BiBloSA","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/1804.00857","atlas_url":"https://app.syntology.ai/?focus=1804.00857","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1804.00857"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/taoshen58/BiBloSA","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"unverified":4},"by_repo_kind":{"official":{"samples":4,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"d4166cae430d786f","entry":"dropout","repo":"taoshen58/BiBloSA","repo_kind":"official","path":"context_fusion/nn.py","file_url":"https://github.com/taoshen58/BiBloSA/blob/HEAD/context_fusion/nn.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"d4166cae430d786f"}},{"code_sha256_prefix":"72aa499a4e4466b2","entry":"expand_tile","repo":"taoshen58/BiBloSA","repo_kind":"official","path":"context_fusion/general.py","file_url":"https://github.com/taoshen58/BiBloSA/blob/HEAD/context_fusion/general.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"72aa499a4e4466b2"}},{"code_sha256_prefix":"e7841940b013046c","entry":"get_initializer","repo":"taoshen58/BiBloSA","repo_kind":"official","path":"context_fusion/general.py","file_url":"https://github.com/taoshen58/BiBloSA/blob/HEAD/context_fusion/general.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e7841940b013046c"}},{"code_sha256_prefix":"ab240d4f661504d0","entry":"get_last_state","repo":"taoshen58/BiBloSA","repo_kind":"official","path":"context_fusion/general.py","file_url":"https://github.com/taoshen58/BiBloSA/blob/HEAD/context_fusion/general.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"ab240d4f661504d0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}