{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/forgetting-transformer-softmax-attention-with","title":"Forgetting Transformer: Softmax Attention with a Forget Gate","arxiv_id":"2503.02130","date":"2025-03-03","proceeding":null,"authors":["Zhixuan Lin","Evgenii Nikishin","Xu Owen He","Aaron Courville"],"abstract":"An essential component of modern recurrent sequence models is the forget gate. While Transformers do not have an explicit recurrent form, we show that a forget gate can be naturally incorporated into Transformers by down-weighting the unnormalized attention scores in a data-dependent way. We name this attention mechanism Forgetting Attention and the resulting model the Forgetting Transformer (FoX). We show that FoX outperforms the Transformer on long-context language modeling, length extrapolation, and short-context downstream tasks, while performing on par with the Transformer on long-context downstream tasks. Moreover, it is compatible with the FlashAttention algorithm and does not require any positional embeddings. Several analyses, including the needle-in-the-haystack test, show that FoX also retains the Transformer's superior long-context capabilities over recurrent sequence models such as Mamba-2, HGRN2, and DeltaNet. We also introduce a \"Pro\" block design that incorporates some common architectural components in recurrent sequence models and find it significantly improves the performance of both FoX and the Transformer. Our code is available at https://github.com/zhixuan-lin/forgetting-transformer.","url_abs":"https://arxiv.org/abs/2503.02130v2","url_pdf":"https://arxiv.org/pdf/2503.02130v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"forgetting-transformer-softmax-attention-with","repo_url":"https://github.com/zhixuan-lin/forgetting-transformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mamba","task_name":"Mamba"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2503.02130","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2503.02130"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/zhixuan-lin/forgetting-transformer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"unverified":11},"by_repo_kind":{"official":{"samples":13,"ran":2,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"bc4bddefd7540078","entry":"elu_p1","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/model/delta_net/delta_net_layer.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/model/delta_net/delta_net_layer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bc4bddefd7540078"}},{"code_sha256_prefix":"e544fdb6373b9f42","entry":"sum_norm","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/model/delta_net/delta_net_layer.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/model/delta_net/delta_net_layer.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e544fdb6373b9f42"}},{"code_sha256_prefix":"f4bcdc1cf181abd3","entry":"constant_schedule","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/schedule/schedule.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/schedule/schedule.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f4bcdc1cf181abd3"}},{"code_sha256_prefix":"47a1ac03e43d6cff","entry":"convert_to_numpy","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/logger.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47a1ac03e43d6cff"}},{"code_sha256_prefix":"df304bdd056129a1","entry":"convert_to_scalar","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/logger.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"df304bdd056129a1"}},{"code_sha256_prefix":"5e23b22ed11c3731","entry":"group_parameters","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/utils.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e23b22ed11c3731"}},{"code_sha256_prefix":"5c43f7270b7b5ef0","entry":"is_power_of_two","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/utils.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5c43f7270b7b5ef0"}},{"code_sha256_prefix":"f474d27c0cab10e5","entry":"is_valid_filename","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/logger.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/logger.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f474d27c0cab10e5"}},{"code_sha256_prefix":"07f244414412b732","entry":"join_schedules","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/schedule/schedule.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/schedule/schedule.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"07f244414412b732"}},{"code_sha256_prefix":"58e6542897e7d424","entry":"maybe_contiguous","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/ops/forgetting_attention.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/ops/forgetting_attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"58e6542897e7d424"}},{"code_sha256_prefix":"46cd56c1ad913051","entry":"one_minus_sqrt_schedule","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/schedule/schedule.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/schedule/schedule.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"46cd56c1ad913051"}},{"code_sha256_prefix":"d0340dda6fca386f","entry":"rounded_multiple","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/ops/forgetting_attention.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/ops/forgetting_attention.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d0340dda6fca386f"}},{"code_sha256_prefix":"e9970b6ad2c69c84","entry":"safe_divide","repo":"zhixuan-lin/forgetting-transformer","repo_kind":"official","path":"src/forgetting_transformer/utils.py","file_url":"https://github.com/zhixuan-lin/forgetting-transformer/blob/HEAD/src/forgetting_transformer/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e9970b6ad2c69c84"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}