{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2604-20920","title":"Simplified Sparse Attention via Gist Tokens","arxiv_id":"2604.20920","date":"2026-04-22","proceeding":null,"authors":["Yuzhen Mao","Michael Y. Li","Emily B. Fox"],"abstract":"Sparse attention can reduce the cost of long-context inference, but most variants introduce new architectural components. We introduce Simplified Sparse Attention (SSA), a simpler approach to sparse attention that requires no architectural changes. Concretely, we first perform continued pretraining on sequences interleaved with gist tokens. We optimize the standard next-token loss as usual, but the gist tokens use an attention mask to restrict what parts of the context the language model can attend to; this teaches the model to pack each chunk's important information into the gist tokens. At inference time, SSA scores chunks via attention between the current query and the small set of gist tokens, selectively unfolding the top-k chunks by reintroducing their corresponding raw tokens. Since the query is scored only against the gist tokens, we avoid the memory-bandwidth cost associated with naive scoring against the full KV cache, without requiring the auxiliary KV cache approach used by sparse attention methods. On LongBench, SSA consistently outperforms compression and inference-time sparse-attention baselines under the same compression ratio. More strikingly, in retrieval-augmented generation, SSA can even outperform full attention after continued pretraining by over 5.7 points. We attribute this to the ability of SSA's selective unfolding, which concentrates attention on the query-relevant chunks and effectively filters out noise. SSA further extends to a hierarchical gist-of-gist variant (H-SSA) that achieves log-linear decoding complexity while maintaining or improving accuracy at high compression ratios up to 32x. The code is available at https://github.com/yuzhenmao/simplified-sparse-attention/.","url_abs":"https://arxiv.org/abs/2604.20920","url_pdf":"https://arxiv.org/pdf/2604.20920","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2604.20920","atlas_url":"https://app.syntology.ai/?focus=2604.20920","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2604.20920"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/yuzhenmao/simplified-sparse-attention","reach":null}],"summary":{"ran":1,"ran_honours":1,"ran_fixture":1,"ran_draft_wrong":2,"unverified":2},"by_repo_kind":{"found_in_text":{"samples":7,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":7,"samples":[{"code_sha256_prefix":"5e8c0fa11169a55a","entry":"_get_select_pg_compiled","repo":"yuzhenmao/simplified-sparse-attention","repo_kind":"found_in_text","path":"attn_candidate.py","file_url":"https://github.com/yuzhenmao/simplified-sparse-attention/blob/HEAD/attn_candidate.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"5e8c0fa11169a55a"}},{"code_sha256_prefix":"a86a0cb421f0cb91","entry":"_next_bucket","repo":"yuzhenmao/simplified-sparse-attention","repo_kind":"found_in_text","path":"attn_candidate.py","file_url":"https://github.com/yuzhenmao/simplified-sparse-attention/blob/HEAD/attn_candidate.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"a86a0cb421f0cb91"}},{"code_sha256_prefix":"818e242de7c7c550","entry":"_pad_last","repo":"yuzhenmao/simplified-sparse-attention","repo_kind":"found_in_text","path":"attn_candidate.py","file_url":"https://github.com/yuzhenmao/simplified-sparse-attention/blob/HEAD/attn_candidate.py","link_basis":"first_harvest_node","language":"python","status":"ran_fixture","verification_level":1,"contract_check":"RAISES","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"818e242de7c7c550"}},{"code_sha256_prefix":"e9c481993fb516ee","entry":"top_k_from_logits","repo":"yuzhenmao/simplified-sparse-attention","repo_kind":"found_in_text","path":"attn_candidate.py","file_url":"https://github.com/yuzhenmao/simplified-sparse-attention/blob/HEAD/attn_candidate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"e9c481993fb516ee"}},{"code_sha256_prefix":"efd034871700fabb","entry":"top_p_from_logits","repo":"yuzhenmao/simplified-sparse-attention","repo_kind":"found_in_text","path":"attn_candidate.py","file_url":"https://github.com/yuzhenmao/simplified-sparse-attention/blob/HEAD/attn_candidate.py","link_basis":"first_harvest_node","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"invariant","behaviour_fingerprint":true,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"efd034871700fabb"}},{"code_sha256_prefix":"37cfc17f67b40418","entry":"_select_per_group","repo":"yuzhenmao/simplified-sparse-attention","repo_kind":"found_in_text","path":"attn_candidate.py","file_url":"https://github.com/yuzhenmao/simplified-sparse-attention/blob/HEAD/attn_candidate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"37cfc17f67b40418"}},{"code_sha256_prefix":"3fc9fd35d8bacef1","entry":"_select_pg_impl","repo":"yuzhenmao/simplified-sparse-attention","repo_kind":"found_in_text","path":"attn_candidate.py","file_url":"https://github.com/yuzhenmao/simplified-sparse-attention/blob/HEAD/attn_candidate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NOASSERTION","inline_ok":false,"mcp_get_code":{"code_sha256":"3fc9fd35d8bacef1"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}