{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/leapformer-enabling-linear-transformers-for","title":"LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions","arxiv_id":"2405.13046","date":"2024-05-18","proceeding":null,"authors":["Victor Agostinelli","Sanghyun Hong","Lizhong Chen"],"abstract":"A promising approach to preserving model performance in linearized transformers is to employ position-based re-weighting functions. However, state-of-the-art re-weighting functions rely heavily on target sequence lengths, making it difficult or impossible to apply them to autoregressive and simultaneous tasks, where the target and sometimes even the input sequence length are unknown. To address this issue, we propose Learned Proportions (LeaP) and LeaPformers. Our contribution is built on two major components. First, we generalize the dependence on explicit positional representations and sequence lengths into dependence on sequence proportions for re-weighting. Second, we replace static positional representations with dynamic proportions derived via a compact module, enabling more flexible attention concentration patterns. We evaluate LeaPformer against eight representative efficient transformers on the Long-Range Arena benchmark, showing that LeaPformer achieves the best quality-throughput trade-off, as well as LeaPformer to Wikitext-103 autoregressive language modeling and simultaneous speech-to-text translation for two language pairs, achieving competitive results.","url_abs":"https://arxiv.org/abs/2405.13046v1","url_pdf":"https://arxiv.org/pdf/2405.13046v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"leapformer-enabling-linear-transformers-for","repo_url":"https://github.com/osu-starlab/leapformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"simultaneous-speech-to-text-translation","task_name":"Simultaneous Speech-to-Text Translation"},{"task_slug":"speech-to-text","task_name":"Speech-to-Text"},{"task_slug":"speech-to-text-translation","task_name":"Speech-to-Text Translation"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2405.13046","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2405.13046"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/osu-starlab/leapformer","reach":null}],"summary":{"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"official":{"samples":4,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"360cfe9d6cecd5d1","entry":"leapformer_attn_train","repo":"osu-starlab/leapformer","repo_kind":"official","path":"fairseq-leapformer/fairseq/modules/leapformer_attention.py","file_url":"https://github.com/osu-starlab/leapformer/blob/HEAD/fairseq-leapformer/fairseq/modules/leapformer_attention.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"360cfe9d6cecd5d1"}},{"code_sha256_prefix":"5d8e94d8a66cea96","entry":"suffix_rundir","repo":"osu-starlab/leapformer","repo_kind":"official","path":"fairseq-leapformer/examples/MMPT/mmpt/utils/load_config.py","file_url":"https://github.com/osu-starlab/leapformer/blob/HEAD/fairseq-leapformer/examples/MMPT/mmpt/utils/load_config.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5d8e94d8a66cea96"}},{"code_sha256_prefix":"d13d24d01f8ac68b","entry":"load_config","repo":"osu-starlab/leapformer","repo_kind":"official","path":"fairseq-leapformer/examples/MMPT/mmpt/utils/load_config.py","file_url":"https://github.com/osu-starlab/leapformer/blob/HEAD/fairseq-leapformer/examples/MMPT/mmpt/utils/load_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d13d24d01f8ac68b"}},{"code_sha256_prefix":"ad1f00f2b92a3b27","entry":"recursive_config","repo":"osu-starlab/leapformer","repo_kind":"official","path":"fairseq-leapformer/examples/MMPT/mmpt/utils/load_config.py","file_url":"https://github.com/osu-starlab/leapformer/blob/HEAD/fairseq-leapformer/examples/MMPT/mmpt/utils/load_config.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ad1f00f2b92a3b27"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}