{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/perceiving-longer-sequences-with-bi","title":"Perceiving Longer Sequences With Bi-Directional Cross-Attention Transformers","arxiv_id":"2402.12138","date":"2024-02-19","proceeding":null,"authors":["Markus Hiller","Krista A. Ehinger","Tom Drummond"],"abstract":"We present a novel bi-directional Transformer architecture (BiXT) which scales linearly with input size in terms of computational cost and memory consumption, but does not suffer the drop in performance or limitation to only one input modality seen with other efficient Transformer-based approaches. BiXT is inspired by the Perceiver architectures but replaces iterative attention with an efficient bi-directional cross-attention module in which input tokens and latent variables attend to each other simultaneously, leveraging a naturally emerging attention-symmetry between the two. This approach unlocks a key bottleneck experienced by Perceiver-like architectures and enables the processing and interpretation of both semantics ('what') and location ('where') to develop alongside each other over multiple layers -- allowing its direct application to dense and instance-based tasks alike. By combining efficiency with the generality and performance of a full Transformer architecture, BiXT can process longer sequences like point clouds, text or images at higher feature resolutions and achieves competitive performance across a range of tasks like point cloud part segmentation, semantic image segmentation, image classification, hierarchical sequence modeling and document retrieval. Our experiments demonstrate that BiXT models outperform larger competitors by leveraging longer sequences more efficiently on vision tasks like classification and segmentation, and perform on par with full Transformer variants on sequence modeling and document retrieval -- but require $28\\%$ fewer FLOPs and are up to $8.4\\times$ faster.","url_abs":"https://arxiv.org/abs/2402.12138v3","url_pdf":"https://arxiv.org/pdf/2402.12138v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"perceiving-longer-sequences-with-bi","repo_url":"https://github.com/mrkshllr/bixt","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"perceiving-longer-sequences-with-bi","repo_url":"https://github.com/axrwl/bidirectional-cross-attention","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":0,"framework":"jax","reach":null}],"tasks":[{"task_slug":"image-classification","task_name":"Image Classification"},{"task_slug":"image-segmentation","task_name":"Image Segmentation"},{"task_slug":"retrieval","task_name":"Retrieval"},{"task_slug":"segmentation","task_name":"Segmentation"},{"task_slug":"semantic-segmentation","task_name":"Semantic Segmentation"},{"task_slug":"image-classification","task_name":"image-classification"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"concatenated-skip-connection","method_name":"Concatenated Skip Connection"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2402.12138","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.12138"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/mrkshllr/bixt","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/axrwl/bidirectional-cross-attention","reach":null}],"summary":{"ran":11,"ran_draft_wrong":3,"ran_fixture":3,"unverified":11},"by_repo_kind":{"official":{"samples":27,"ran":16,"repositories":1},"listed":{"samples":1,"ran":1,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":27,"samples":[{"code_sha256_prefix":"3809092d302a14aa","entry":"BidirectionalCrossAttention","repo":"axrwl/bidirectional-cross-attention","repo_kind":"listed","path":"bidirectional_cross_attention_jax.py","file_url":"https://github.com/axrwl/bidirectional-cross-attention/blob/HEAD/bidirectional_cross_attention_jax.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3809092d302a14aa"}},{"code_sha256_prefix":"406496ff28d67b77","entry":"CrossAttention","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"406496ff28d67b77"}},{"code_sha256_prefix":"25760af757f9bb12","entry":"CrossAttentionOneSided","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"25760af757f9bb12"}},{"code_sha256_prefix":"59f48e4236839efb","entry":"DropPath","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"59f48e4236839efb"}},{"code_sha256_prefix":"fa167a84fdd9edea","entry":"Mlp","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"fa167a84fdd9edea"}},{"code_sha256_prefix":"20509c2dd02760b3","entry":"PatchEmbed","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"20509c2dd02760b3"}},{"code_sha256_prefix":"d5590e86670ba51d","entry":"PatchEmbedHalfStride","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d5590e86670ba51d"}},{"code_sha256_prefix":"8b845e4a681b8a0a","entry":"PatchEmbedQuarterStride","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8b845e4a681b8a0a"}},{"code_sha256_prefix":"5a318973aa4ca2b4","entry":"PositionalEncodingFourierAdpt","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5a318973aa4ca2b4"}},{"code_sha256_prefix":"24c1740f65afedc7","entry":"PositionalEncodingFourierXCiTEff","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"24c1740f65afedc7"}},{"code_sha256_prefix":"82dbf42759abac64","entry":"SelfAttention","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"82dbf42759abac64"}},{"code_sha256_prefix":"65b709c708e72249","entry":"_ntuple","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"65b709c708e72249"}},{"code_sha256_prefix":"119ce2c5455542b0","entry":"_trunc_normal_","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"119ce2c5455542b0"}},{"code_sha256_prefix":"5cb5ef5a1ebc83fa","entry":"named_apply","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5cb5ef5a1ebc83fa"}},{"code_sha256_prefix":"214d4d32ef0c69b0","entry":"pop_posenc_pos_ft384","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"214d4d32ef0c69b0"}},{"code_sha256_prefix":"c92dcfe50ad922d8","entry":"trunc_normal_","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"c92dcfe50ad922d8"}},{"code_sha256_prefix":"12a0192165e48bca","entry":"trunc_normal_tf_","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"12a0192165e48bca"}},{"code_sha256_prefix":"f09b3147fa564d80","entry":"BiXT","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"f09b3147fa564d80"}},{"code_sha256_prefix":"7ff9d17c8bb3e756","entry":"BiXTBlock","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"7ff9d17c8bb3e756"}},{"code_sha256_prefix":"a787a84303d152b5","entry":"CABlock","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a787a84303d152b5"}},{"code_sha256_prefix":"ace5c250f412796c","entry":"CAOneSidedBlock","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"ace5c250f412796c"}},{"code_sha256_prefix":"cdadaea5ce13ec84","entry":"SABlock","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"cdadaea5ce13ec84"}},{"code_sha256_prefix":"de9516aa3ff5eb1d","entry":"get_init_weights_bixt","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"de9516aa3ff5eb1d"}},{"code_sha256_prefix":"dcff31ee17975b70","entry":"init_weights_bixt_jax","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"dcff31ee17975b70"}},{"code_sha256_prefix":"e5f05db1be8f79d9","entry":"init_weights_bixt_moco","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e5f05db1be8f79d9"}},{"code_sha256_prefix":"d30ca1920d7a4d23","entry":"init_weights_bixt_timm","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"d30ca1920d7a4d23"}},{"code_sha256_prefix":"9e505b05709a5466","entry":"lecun_normal_","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"9e505b05709a5466"}},{"code_sha256_prefix":"8c3be2a32f59fc2a","entry":"variance_scaling_","repo":"mrkshllr/bixt","repo_kind":"official","path":"timm/models/bixt.py","file_url":"https://github.com/mrkshllr/bixt/blob/HEAD/timm/models/bixt.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"8c3be2a32f59fc2a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}