{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/just-read-twice-closing-the-recall-gap-for","title":"Just read twice: closing the recall gap for recurrent language models","arxiv_id":"2407.05483","date":"2024-07-07","proceeding":null,"authors":["Simran Arora","Aman Timalsina","Aaryan Singhal","Benjamin Spector","Sabri Eyuboglu","Xinyi Zhao","Ashish Rao","Atri Rudra","Christopher Ré"],"abstract":"Recurrent large language models that compete with Transformers in language modeling perplexity are emerging at a rapid rate (e.g., Mamba, RWKV). Excitingly, these architectures use a constant amount of memory during inference. However, due to the limited memory, recurrent LMs cannot recall and use all the information in long contexts leading to brittle in-context learning (ICL) quality. A key challenge for efficient LMs is selecting what information to store versus discard. In this work, we observe the order in which information is shown to the LM impacts the selection difficulty. To formalize this, we show that the hardness of information recall reduces to the hardness of a problem called set disjointness (SD), a quintessential problem in communication complexity that requires a streaming algorithm (e.g., recurrent model) to decide whether inputted sets are disjoint. We empirically and theoretically show that the recurrent memory required to solve SD changes with set order, i.e., whether the smaller set appears first in-context. Our analysis suggests, to mitigate the reliance on data order, we can put information in the right order in-context or process prompts non-causally. Towards that end, we propose: (1) JRT-Prompt, where context gets repeated multiple times in the prompt, effectively showing the model all data orders. This gives $11.0 \\pm 1.3$ points of improvement, averaged across $16$ recurrent LMs and the $6$ ICL tasks, with $11.9\\times$ higher throughput than FlashAttention-2 for generation prefill (length $32$k, batch size $16$, NVidia H100). We then propose (2) JRT-RNN, which uses non-causal prefix-linear-attention to process prompts and provides $99\\%$ of Transformer quality at $360$M params., $30$B tokens and $96\\%$ at $1.3$B params., $50$B tokens on average across the tasks, with $19.2\\times$ higher throughput for prefill than FA2.","url_abs":"https://arxiv.org/abs/2407.05483v1","url_pdf":"https://arxiv.org/pdf/2407.05483v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"just-read-twice-closing-the-recall-gap-for","repo_url":"https://github.com/HazyResearch/prefix-linear-attention","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok"}}],"tasks":[{"task_slug":"in-context-learning","task_name":"In-Context Learning"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"mamba","task_name":"Mamba"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"set","method_name":"SET"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2407.05483","atlas_url":"https://app.syntology.ai/?focus=2407.05483","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2407.05483"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/HazyResearch/prefix-linear-attention","reach":{"status":"ok"}}],"summary":{"ran":3,"unverified":2},"by_repo_kind":{"official":{"samples":5,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":5,"samples":[{"code_sha256_prefix":"1586b639a5abd0b0","entry":"init_to_zero","repo":"HazyResearch/prefix-linear-attention","repo_kind":"official","path":"based/ops/triton/linear.py","file_url":"https://github.com/HazyResearch/prefix-linear-attention/blob/HEAD/based/ops/triton/linear.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1586b639a5abd0b0"}},{"code_sha256_prefix":"c6283b7a1a1785ba","entry":"maybe_align","repo":"HazyResearch/prefix-linear-attention","repo_kind":"official","path":"based/ops/layer_norm.py","file_url":"https://github.com/HazyResearch/prefix-linear-attention/blob/HEAD/based/ops/layer_norm.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"c6283b7a1a1785ba"}},{"code_sha256_prefix":"0e11bb8b11afc5b8","entry":"sample","repo":"HazyResearch/prefix-linear-attention","repo_kind":"official","path":"based/generation.py","file_url":"https://github.com/HazyResearch/prefix-linear-attention/blob/HEAD/based/generation.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"0e11bb8b11afc5b8"}},{"code_sha256_prefix":"26116b1ac7fbe421","entry":"decode","repo":"HazyResearch/prefix-linear-attention","repo_kind":"official","path":"based/generation.py","file_url":"https://github.com/HazyResearch/prefix-linear-attention/blob/HEAD/based/generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"26116b1ac7fbe421"}},{"code_sha256_prefix":"a8c6d95092ae59ba","entry":"sample_speculative","repo":"HazyResearch/prefix-linear-attention","repo_kind":"official","path":"based/generation.py","file_url":"https://github.com/HazyResearch/prefix-linear-attention/blob/HEAD/based/generation.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"a8c6d95092ae59ba"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}