{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/on-the-computational-power-of-transformers","title":"On the Computational Power of Transformers and its Implications in Sequence Modeling","arxiv_id":"2006.09286","date":"2020-06-16","proceeding":"CONLL 2020","authors":["Satwik Bhattamishra","Arkil Patel","Navin Goyal"],"abstract":"Transformers are being used extensively across several sequence modeling tasks. Significant research effort has been devoted to experimentally probe the inner workings of Transformers. However, our conceptual and theoretical understanding of their power and inherent limitations is still nascent. In particular, the roles of various components in Transformers such as positional encodings, attention heads, residual connections, and feedforward networks, are not clear. In this paper, we take a step towards answering these questions. We analyze the computational power as captured by Turing-completeness. We first provide an alternate and simpler proof to show that vanilla Transformers are Turing-complete and then we prove that Transformers with only positional masking and without any positional encoding are also Turing-complete. We further analyze the necessity of each component for the Turing-completeness of the network; interestingly, we find that a particular type of residual connection is necessary. We demonstrate the practical implications of our results via experiments on machine translation and synthetic tasks.","url_abs":"https://arxiv.org/abs/2006.09286v3","url_pdf":"https://arxiv.org/pdf/2006.09286v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"on-the-computational-power-of-transformers","repo_url":"https://github.com/satwik77/Transformer-Computation-Analysis","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"residual-connection","method_name":"Residual Connection"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2006.09286","atlas_url":"https://app.syntology.ai/?focus=2006.09286","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.09286"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/satwik77/Transformer-Computation-Analysis","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran_draft_wrong":2,"ran_violates":1,"unverified":12},"by_repo_kind":{"official":{"samples":15,"ran":3,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"20bb0ff4a2d77a03","entry":"attention","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/components/self_attention.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/components/self_attention.py","link_basis":"plan_row","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"20bb0ff4a2d77a03"}},{"code_sha256_prefix":"a3722169bbc81569","entry":"clones","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/components/utils.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/components/utils.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a3722169bbc81569"}},{"code_sha256_prefix":"66fbf162663d4cff","entry":"subsequent_mask","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/components/utils.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/components/utils.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"66fbf162663d4cff"}},{"code_sha256_prefix":"34e826973176fc38","entry":"cal_score","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/utils/evaluate.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/utils/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"34e826973176fc38"}},{"code_sha256_prefix":"74e35c9e284520a1","entry":"cal_score_AP","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/utils/evaluate.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/utils/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"74e35c9e284520a1"}},{"code_sha256_prefix":"ae3b0a421287c5c9","entry":"compute_bleu","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/utils/bleu.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/utils/bleu.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ae3b0a421287c5c9"}},{"code_sha256_prefix":"ebe3d1e68daeb780","entry":"get_logger","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/utils/logger.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/utils/logger.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ebe3d1e68daeb780"}},{"code_sha256_prefix":"c930b9e88eed3346","entry":"get_std_opt","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/utils/helper.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/utils/helper.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c930b9e88eed3346"}},{"code_sha256_prefix":"5f0fe04ab8eb68da","entry":"greedy_decode","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/components/utils.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/components/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"5f0fe04ab8eb68da"}},{"code_sha256_prefix":"a4731b662e3d1cd8","entry":"make_model","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/model.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"a4731b662e3d1cd8"}},{"code_sha256_prefix":"c179d23a93633b29","entry":"run_epoch","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/model.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/model.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"c179d23a93633b29"}},{"code_sha256_prefix":"543b4d33c65c0c61","entry":"sent_to_idx","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/utils/sentence_processing.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/utils/sentence_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"543b4d33c65c0c61"}},{"code_sha256_prefix":"0a8dfa17ef60febe","entry":"sent_to_tensor","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/utils/sentence_processing.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/utils/sentence_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"0a8dfa17ef60febe"}},{"code_sha256_prefix":"6a992d9f1625b3ae","entry":"sents_to_idx","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/utils/sentence_processing.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/utils/sentence_processing.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a992d9f1625b3ae"}},{"code_sha256_prefix":"6d3d89720d9d4bd7","entry":"stack_to_string","repo":"satwik77/Transformer-Computation-Analysis","repo_kind":"official","path":"Transformer/src/utils/evaluate.py","file_url":"https://github.com/satwik77/Transformer-Computation-Analysis/blob/HEAD/Transformer/src/utils/evaluate.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6d3d89720d9d4bd7"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}