{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/repeat-after-me-transformers-are-better-than","title":"Repeat After Me: Transformers are Better than State Space Models at Copying","arxiv_id":"2402.01032","date":"2024-02-01","proceeding":null,"authors":["Samy Jelassi","David Brandfonbrener","Sham M. Kakade","Eran Malach"],"abstract":"Transformers are the dominant architecture for sequence modeling, but there is growing interest in models that use a fixed-size latent state that does not depend on the sequence length, which we refer to as \"generalized state space models\" (GSSMs). In this paper we show that while GSSMs are promising in terms of inference-time efficiency, they are limited compared to transformer models on tasks that require copying from the input context. We start with a theoretical analysis of the simple task of string copying and prove that a two layer transformer can copy strings of exponential length while GSSMs are fundamentally limited by their fixed-size latent state. Empirically, we find that transformers outperform GSSMs in terms of efficiency and generalization on synthetic tasks that require copying the context. Finally, we evaluate pretrained large language models and find that transformer models dramatically outperform state space models at copying and retrieving information from context. Taken together, these results suggest a fundamental gap between transformers and GSSMs on tasks of practical interest.","url_abs":"https://arxiv.org/abs/2402.01032v2","url_pdf":"https://arxiv.org/pdf/2402.01032v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"repeat-after-me-transformers-are-better-than","repo_url":"https://github.com/sjelassi/transformers_ssm_copy","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"repeat-after-me-transformers-are-better-than","repo_url":"https://github.com/hazyresearch/based","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null}],"tasks":[{"task_slug":"state-space-models","task_name":"State Space Models"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2402.01032","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2402.01032"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sjelassi/transformers_ssm_copy","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/hazyresearch/based","reach":null}],"summary":{"ran":10},"by_repo_kind":{"official":{"samples":10,"ran":10,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"0d92dcc623352c2c","entry":"arr_to_str","repo":"sjelassi/transformers_ssm_copy","repo_kind":"official","path":"synthetic_exps/data_utils.py","file_url":"https://github.com/sjelassi/transformers_ssm_copy/blob/HEAD/synthetic_exps/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0d92dcc623352c2c"}},{"code_sha256_prefix":"f25aff9e401bdd61","entry":"build_mpt_alibi_tensor","repo":"sjelassi/transformers_ssm_copy","repo_kind":"official","path":"synthetic_exps/models/alibi_model.py","file_url":"https://github.com/sjelassi/transformers_ssm_copy/blob/HEAD/synthetic_exps/models/alibi_model.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f25aff9e401bdd61"}},{"code_sha256_prefix":"b9b929da44e45018","entry":"ce_loss","repo":"sjelassi/transformers_ssm_copy","repo_kind":"official","path":"synthetic_exps/train_utils.py","file_url":"https://github.com/sjelassi/transformers_ssm_copy/blob/HEAD/synthetic_exps/train_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b9b929da44e45018"}},{"code_sha256_prefix":"72642a4d8a990cb5","entry":"custom_get_scheduler","repo":"sjelassi/transformers_ssm_copy","repo_kind":"official","path":"synthetic_exps/train_utils.py","file_url":"https://github.com/sjelassi/transformers_ssm_copy/blob/HEAD/synthetic_exps/train_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"72642a4d8a990cb5"}},{"code_sha256_prefix":"67b1c1bbca4316ab","entry":"exact_match_score","repo":"sjelassi/transformers_ssm_copy","repo_kind":"official","path":"pretrained_exps/qa_evaluation_utils.py","file_url":"https://github.com/sjelassi/transformers_ssm_copy/blob/HEAD/pretrained_exps/qa_evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"67b1c1bbca4316ab"}},{"code_sha256_prefix":"933e94e11554ccb1","entry":"f1_score","repo":"sjelassi/transformers_ssm_copy","repo_kind":"official","path":"pretrained_exps/qa_evaluation_utils.py","file_url":"https://github.com/sjelassi/transformers_ssm_copy/blob/HEAD/pretrained_exps/qa_evaluation_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"933e94e11554ccb1"}},{"code_sha256_prefix":"0314589d134038a1","entry":"generate_str_unique_ngram","repo":"sjelassi/transformers_ssm_copy","repo_kind":"official","path":"synthetic_exps/data_utils.py","file_url":"https://github.com/sjelassi/transformers_ssm_copy/blob/HEAD/synthetic_exps/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0314589d134038a1"}},{"code_sha256_prefix":"32a6a2b055feb482","entry":"get_optimizer","repo":"sjelassi/transformers_ssm_copy","repo_kind":"official","path":"synthetic_exps/train_utils.py","file_url":"https://github.com/sjelassi/transformers_ssm_copy/blob/HEAD/synthetic_exps/train_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"32a6a2b055feb482"}},{"code_sha256_prefix":"c29fc928dae85749","entry":"normalize_answer","repo":"sjelassi/transformers_ssm_copy","repo_kind":"official","path":"pretrained_exps/qa_evaluation_utils.py","file_url":"https://github.com/sjelassi/transformers_ssm_copy/blob/HEAD/pretrained_exps/qa_evaluation_utils.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c29fc928dae85749"}},{"code_sha256_prefix":"b80507bc684ce720","entry":"rand_num","repo":"sjelassi/transformers_ssm_copy","repo_kind":"official","path":"synthetic_exps/data_utils.py","file_url":"https://github.com/sjelassi/transformers_ssm_copy/blob/HEAD/synthetic_exps/data_utils.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"b80507bc684ce720"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}