{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/born-a-transformer-always-a-transformer","title":"Born a Transformer -- Always a Transformer?","arxiv_id":"2505.21785","date":"2025-05-27","proceeding":null,"authors":["Yana Veitsman","Mayank Jobanputra","Yash Sarrof","Aleksandra Bakalova","Vera Demberg","Ellie Pavlick","Michael Hahn"],"abstract":"Transformers have theoretical limitations in modeling certain sequence-to-sequence tasks, yet it remains largely unclear if these limitations play a role in large-scale pretrained LLMs, or whether LLMs might effectively overcome these constraints in practice due to the scale of both the models themselves and their pretraining data. We explore how these architectural constraints manifest after pretraining, by studying a family of $\\textit{retrieval}$ and $\\textit{copying}$ tasks inspired by Liu et al. [2024a]. We use a recently proposed framework for studying length generalization [Huang et al., 2025] to provide guarantees for each of our settings. Empirically, we observe an $\\textit{induction-versus-anti-induction}$ asymmetry, where pretrained models are better at retrieving tokens to the right (induction) rather than the left (anti-induction) of a query token. This asymmetry disappears upon targeted fine-tuning if length-generalization is guaranteed by theory. Mechanistic analysis reveals that this asymmetry is connected to the differences in the strength of induction versus anti-induction circuits within pretrained transformers. We validate our findings through practical experiments on real-world tasks demonstrating reliability risks. Our results highlight that pretraining selectively enhances certain transformer capabilities, but does not overcome fundamental length-generalization limits.","url_abs":"https://arxiv.org/abs/2505.21785v2","url_pdf":"https://arxiv.org/pdf/2505.21785v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"born-a-transformer-always-a-transformer","repo_url":"https://github.com/lacoco-lab/always_a_transformer","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":0,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=2505.21785","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2505.21785"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/lacoco-lab/always_a_transformer","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":12},"by_repo_kind":{"official":{"samples":12,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"8e508289d57d70e2","entry":"compute_means_stds","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"copying/evaluation.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/copying/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"8e508289d57d70e2"}},{"code_sha256_prefix":"fbf6b639efd24db3","entry":"eval_collator","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"finetuning/copy_str/finetune_copy.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/finetuning/copy_str/finetune_copy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbf6b639efd24db3"}},{"code_sha256_prefix":"68af3b0107b96e33","entry":"find_adjacent_characters","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"finetuning/inductionhead/generate_data.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/finetuning/inductionhead/generate_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"68af3b0107b96e33"}},{"code_sha256_prefix":"3270771efda2e25f","entry":"generate_string","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"finetuning/inductionhead/generate_data.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/finetuning/inductionhead/generate_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"3270771efda2e25f"}},{"code_sha256_prefix":"f1360f0d8bfa8542","entry":"generate_valid_sample","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"finetuning/inductionhead/generate_data.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/finetuning/inductionhead/generate_data.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"f1360f0d8bfa8542"}},{"code_sha256_prefix":"478f3d6762fd9674","entry":"load_adv_accuracies","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"visualisations/finetune_ood.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/visualisations/finetune_ood.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"478f3d6762fd9674"}},{"code_sha256_prefix":"ac77a3c20a83e890","entry":"load_all_models","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"copying/evaluation.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/copying/evaluation.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac77a3c20a83e890"}},{"code_sha256_prefix":"d2f2fdec627fb1b5","entry":"load_and_prepare_data","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"finetuning/flipflop/finetune_flipflop.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/finetuning/flipflop/finetune_flipflop.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d2f2fdec627fb1b5"}},{"code_sha256_prefix":"a9ad18c47c5a8531","entry":"load_data_from_file","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"finetuning/copy_str/finetune_copy.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/finetuning/copy_str/finetune_copy.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a9ad18c47c5a8531"}},{"code_sha256_prefix":"ac35274d97750578","entry":"load_data_from_file","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"finetuning/flipflop/finetune_flipflop.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/finetuning/flipflop/finetune_flipflop.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ac35274d97750578"}},{"code_sha256_prefix":"5e41c7d3af1343c9","entry":"read_accuracy","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"visualisations/finetune_ood.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/visualisations/finetune_ood.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5e41c7d3af1343c9"}},{"code_sha256_prefix":"fe46d9a5ad18a011","entry":"task_from_filename","repo":"lacoco-lab/always_a_transformer","repo_kind":"official","path":"visualisations/finetune_ood.py","file_url":"https://github.com/lacoco-lab/always_a_transformer/blob/HEAD/visualisations/finetune_ood.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fe46d9a5ad18a011"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}