{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/linear-transformers-with-learnable-kernel","title":"Linear Transformers with Learnable Kernel Functions are Better In-Context Models","arxiv_id":"2402.10644","date":"2024-02-16","proceeding":null,"authors":["Yaroslav Aksenov","Nikita Balagansky","Sofia Maria Lo Cicero Vaina","Boris Shaposhnikov","Alexey Gorbatovski","Daniil Gavrilov"],"abstract":"Advancing the frontier of subquadratic architectures for Language Models (LMs) is crucial in the rapidly evolving field of natural language processing. Current innovations, including State Space Models, were initially celebrated for surpassing Transformer performance on language modeling tasks. However, these models have revealed deficiencies in essential In-Context Learning capabilities - a domain where the Transformer traditionally shines. The Based model emerged as a hybrid solution, blending a Linear Transformer with a kernel inspired by the Taylor expansion of exponential functions, augmented by convolutional networks. Mirroring the Transformer's in-context adeptness, it became a strong contender in the field. In our work, we present a singular, elegant alteration to the Based kernel that amplifies its In-Context Learning abilities evaluated with the Multi-Query Associative Recall task and overall language modeling process, as demonstrated on the Pile dataset.","url_abs":"https://arxiv.org/abs/2402.10644v2","url_pdf":"https://arxiv.org/pdf/2402.10644v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"linear-transformers-with-learnable-kernel","repo_url":"https://github.com/sustcsonglin/flash-linear-attention","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"linear-transformers-with-learnable-kernel","repo_url":"https://github.com/corl-team/rebased","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"Apache-2.0"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2402.10644","atlas_url":"https://app.syntology.ai/?focus=2402.10644","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.10644"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sustcsonglin/flash-linear-attention","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/corl-team/rebased","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":6},"by_repo_kind":{"official":{"samples":6,"ran":6,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8def605349357890","entry":"fft_conv","repo":"corl-team/rebased","repo_kind":"official","path":"src/mixers/convolution.py","file_url":"https://github.com/corl-team/rebased/blob/HEAD/src/mixers/convolution.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"8def605349357890"}},{"code_sha256_prefix":"c849028785a49d49","entry":"import_from_str","repo":"corl-team/rebased","repo_kind":"official","path":"src/utils.py","file_url":"https://github.com/corl-team/rebased/blob/HEAD/src/utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"c849028785a49d49"}},{"code_sha256_prefix":"684a4794c1e731b4","entry":"init_feature_map","repo":"corl-team/rebased","repo_kind":"official","path":"src/mixers/rebased.py","file_url":"https://github.com/corl-team/rebased/blob/HEAD/src/mixers/rebased.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"684a4794c1e731b4"}},{"code_sha256_prefix":"e8f74b2adeb59c06","entry":"init_feature_map","repo":"corl-team/rebased","repo_kind":"official","path":"flash_linear_attention/fla/layers/rebased.py","file_url":"https://github.com/corl-team/rebased/blob/HEAD/flash_linear_attention/fla/layers/rebased.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"e8f74b2adeb59c06"}},{"code_sha256_prefix":"2101df8aafbd7a12","entry":"init_feature_map","repo":"corl-team/rebased","repo_kind":"official","path":"flash_linear_attention/fla/layers/rebased_fast.py","file_url":"https://github.com/corl-team/rebased/blob/HEAD/flash_linear_attention/fla/layers/rebased_fast.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"2101df8aafbd7a12"}},{"code_sha256_prefix":"b91fbb3bff1255fb","entry":"init_feature_map","repo":"corl-team/rebased","repo_kind":"official","path":"src/mixers/based.py","file_url":"https://github.com/corl-team/rebased/blob/HEAD/src/mixers/based.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"b91fbb3bff1255fb"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}