{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/arxiv-2607-22361","title":"Indexing: the Beginning and the End","arxiv_id":"2607.22361","date":"2026-07-24","proceeding":null,"authors":["Alexander Kozachinskiy","Vicente Opazo","Felipe Urrutia"],"abstract":"We study information bottlenecks in modern deep-learning architectures -- RNNs, softmax transformers, linear-attention transformers and state-space models -- through the lens of the indexing primitive. In this primitive, the input consists of $n$ bits and one integer $i$ from $1$ to $n$ called the index, and the output equals the value of the $i$-th bit. We introduce causal complexity for masked architectures. We show that architectures with low causal complexity cannot solve the indexing primitive in any constant number of layers when the index appears at the end of the input. In particular, this limitation applies to low-parameter RNNs, SSMs and masked linear-attention transformers. In contrast, small softmax transformers can solve it in one layer, while non-masked linear-attention transformers can solve it in 2, which separates them from their masked counterparts. In turn, when the index appears at the beginning, we show that small RNNs are capable of solving this task in 1 layer, while all the other architectures require 2. All our impossibility results are unconditional and apply even to models that employ infinite-precision real arithmetic. Moreover, experiments for up to $n=64$ qualitatively align with our theory: configurations with low-parameter theoretical solutions learn the indexing task easily, while configurations that do not admit such theoretical solutions struggle to learn as the sequence length grows.","url_abs":"https://arxiv.org/abs/2607.22361","url_pdf":"https://arxiv.org/pdf/2607.22361","source":{"archive":null,"snapshot":"2025-07-28","note":"not in the Papers with Code archive (frozen at the snapshot)","row_kind":"graph","title_abstract_authors_date":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)"},"code_links":[],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2607.22361","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2607.22361"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"mentioned_in_github":null,"is_official":null,"provenance":"deterministic:regex_extraction","mentioned_in_paper":null,"url":"https://github.com/visho33/IndexRetrieval","reach":null}],"summary":{"ran":5,"unverified":1},"by_repo_kind":{"found_in_text":{"samples":6,"ran":5,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":6,"samples":[{"code_sha256_prefix":"f1ec3c2f7a156bfa","entry":"as_bool","repo":"visho33/IndexRetrieval","repo_kind":"found_in_text","path":"tables.py","file_url":"https://github.com/visho33/IndexRetrieval/blob/HEAD/tables.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"f1ec3c2f7a156bfa"}},{"code_sha256_prefix":"2f6ba00c515f037a","entry":"as_float","repo":"visho33/IndexRetrieval","repo_kind":"found_in_text","path":"tables.py","file_url":"https://github.com/visho33/IndexRetrieval/blob/HEAD/tables.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"2f6ba00c515f037a"}},{"code_sha256_prefix":"711e77cf91407f79","entry":"make_batch","repo":"visho33/IndexRetrieval","repo_kind":"found_in_text","path":"data.py","file_url":"https://github.com/visho33/IndexRetrieval/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"711e77cf91407f79"}},{"code_sha256_prefix":"5ec4dbc58de6aaed","entry":"make_dataloader","repo":"visho33/IndexRetrieval","repo_kind":"found_in_text","path":"data.py","file_url":"https://github.com/visho33/IndexRetrieval/blob/HEAD/data.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"5ec4dbc58de6aaed"}},{"code_sha256_prefix":"44a2c88072d3b789","entry":"read_csv","repo":"visho33/IndexRetrieval","repo_kind":"found_in_text","path":"tables.py","file_url":"https://github.com/visho33/IndexRetrieval/blob/HEAD/tables.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"44a2c88072d3b789"}},{"code_sha256_prefix":"1cb83d460a28156f","entry":"make_position_encoding","repo":"visho33/IndexRetrieval","repo_kind":"found_in_text","path":"model.py","file_url":"https://github.com/visho33/IndexRetrieval/blob/HEAD/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1cb83d460a28156f"}}]},"arxiv_metadata":{"licence":"arXiv metadata, CC0 1.0 (https://info.arxiv.org/help/license)","fields":["title","abstract","authors","date"],"primary_category":"cs.LG","source":"arxiv_2026.jsonl"},"syntology_extracted_results":null}