{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2605-20670","title":"LT2: Linear-Time Looped Transformers","arxiv_id":"2605.20670","date":"2026-05-20","proceeding":null,"authors":["Chunyuan Deng","Yizhe Zhang","Rui-Jie Zhu","Yuanyuan Xu","Jiarui Liu","T. S. Eugene Ng","Hanjie Chen"],"abstract":"Looped Transformers (LT) have emerged as a powerful architecture by iterating their layers multiple times before decoding the final token. However, pairing them with full attention retains quadratic complexity, making them computationally expensive and slow. We introduce LT2 (Linear-Time Looped Transformers), a family of looped architectures that replace quadratic softmax attention with subquadratic, linear-time attention. We study two variants: LT2-linear with linear attention and LT2-sparse with sparse attention. We find that looping uniquely synergizes with these variants: it enables iterative memory refinement in linear attention and progressively expands the effective receptive field in sparse attention. We formalize these benefits theoretically and demonstrate consistent empirical gains across controlled recall, state-tracking, and language modeling tasks. We then explore LT2-hybrid, which combines different attention variants in a looped setting. Two variants are especially promising: LT2-hybrid (GDN+DSA), which interleaves linear and sparse attention to maximize efficiency and matches the standard looped transformer's quality at fully linear-time cost; and LT2-hybrid (Full+GDN), which interleaves GDN with a small fraction of full attention layers to maximize quality, surpassing the standard looped transformer in both performance and efficiency. We also show how to convert a pre-trained LT into an LT2-hybrid model. With about 1B tokens of training, our converted model, Ouro-hybrid-1.4B, outperforms industry-level 1B models and is competitive with industry-level 4B models while retaining the speed benefits of linear-time attention. Together, these results show a clear path toward making looped transformers more scalable and advancing efficient, capable small language models.","url_abs":"https://arxiv.org/abs/2605.20670","url_pdf":"https://arxiv.org/pdf/2605.20670","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2605.20670","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2605.20670"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/chili-lab/LT2","reach":null}],"summary":{"ran":13,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":15,"ran":13,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"26ff74d6538ab161","entry":"attention_flops_per_token","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"apps/LT2/transformer.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/apps/LT2/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"26ff74d6538ab161"}},{"code_sha256_prefix":"5552f9c3667da10b","entry":"compute_correlation","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"apps/LT2/kernel/dsa/fp8_lighting_indexer.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/apps/LT2/kernel/dsa/fp8_lighting_indexer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5552f9c3667da10b"}},{"code_sha256_prefix":"08410d4120383939","entry":"create_slurm_script","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"submit_eval_jobs.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/submit_eval_jobs.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"08410d4120383939"}},{"code_sha256_prefix":"e0ebc4027f1bd03b","entry":"dataclass_from_dict","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"lingua/args.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/lingua/args.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"e0ebc4027f1bd03b"}},{"code_sha256_prefix":"116b2f19f9a6148c","entry":"dataclass_to_dict","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"lingua/args.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/lingua/args.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"116b2f19f9a6148c"}},{"code_sha256_prefix":"03ebb650056220da","entry":"discover_checkpoints","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"upload_to_hf.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/upload_to_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"03ebb650056220da"}},{"code_sha256_prefix":"df220aefba133398","entry":"distribute_data_to_rank","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"lingua/data.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/lingua/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"df220aefba133398"}},{"code_sha256_prefix":"8c82e41cb936a35d","entry":"find_and_sanitize_chunks","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"lingua/data.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/lingua/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"8c82e41cb936a35d"}},{"code_sha256_prefix":"15be2e56e073a228","entry":"flatten_dict","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"lingua/args.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/lingua/args.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"15be2e56e073a228"}},{"code_sha256_prefix":"bfdcca22e67a0a67","entry":"get_empty_buffer_state","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"lingua/data.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/lingua/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"bfdcca22e67a0a67"}},{"code_sha256_prefix":"5e36da6fe12b04f3","entry":"get_num_flop_per_token","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"apps/LT2/transformer.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/apps/LT2/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"5e36da6fe12b04f3"}},{"code_sha256_prefix":"04f070ff6d753818","entry":"step_to_tokens_label","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"upload_to_hf.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/upload_to_hf.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"04f070ff6d753818"}},{"code_sha256_prefix":"0cfebed388443ee6","entry":"validate_tensor_match","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"apps/LT2/kernel/dsa/fp8_lighting_indexer.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/apps/LT2/kernel/dsa/fp8_lighting_indexer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"0cfebed388443ee6"}},{"code_sha256_prefix":"a11baf6e8cca859b","entry":"consolidate_checkpoints","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"lingua/checkpoint.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/lingua/checkpoint.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"a11baf6e8cca859b"}},{"code_sha256_prefix":"3e67867b7d7bdcd0","entry":"ref_sparse_mla_fwd_interface","repo":"chili-lab/LT2","repo_kind":"found_in_text","path":"apps/LT2/kernel/dsa/sparse_mla_fwd.py","file_url":"https://github.com/chili-lab/LT2/blob/HEAD/apps/LT2/kernel/dsa/sparse_mla_fwd.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"BSD-3-Clause","inline_ok":true,"mcp_get_code":{"code_sha256":"3e67867b7d7bdcd0"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}