{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/universal-transformers","title":"Universal Transformers","arxiv_id":"1807.03819","date":"2018-07-10","proceeding":"ICLR 2019 5","authors":["Mostafa Dehghani","Stephan Gouws","Oriol Vinyals","Jakob Uszkoreit","Łukasz Kaiser"],"abstract":"Recurrent neural networks (RNNs) sequentially process data by updating their\nstate with each new data point, and have long been the de facto choice for\nsequence modeling tasks. However, their inherently sequential computation makes\nthem slow to train. Feed-forward and convolutional architectures have recently\nbeen shown to achieve superior results on some sequence modeling tasks such as\nmachine translation, with the added advantage that they concurrently process\nall inputs in the sequence, leading to easy parallelization and faster training\ntimes. Despite these successes, however, popular feed-forward sequence models\nlike the Transformer fail to generalize in many simple tasks that recurrent\nmodels handle with ease, e.g. copying strings or even simple logical inference\nwhen the string or formula lengths exceed those observed at training time. We\npropose the Universal Transformer (UT), a parallel-in-time self-attentive\nrecurrent sequence model which can be cast as a generalization of the\nTransformer model and which addresses these issues. UTs combine the\nparallelizability and global receptive field of feed-forward sequence models\nlike the Transformer with the recurrent inductive bias of RNNs. We also add a\ndynamic per-position halting mechanism and find that it improves accuracy on\nseveral tasks. In contrast to the standard Transformer, under certain\nassumptions, UTs can be shown to be Turing-complete. Our experiments show that\nUTs outperform standard Transformers on a wide range of algorithmic and\nlanguage understanding tasks, including the challenging LAMBADA language\nmodeling task where UTs achieve a new state of the art, and machine translation\nwhere UTs achieve a 0.9 BLEU improvement over Transformers on the WMT14 En-De\ndataset.","url_abs":"http://arxiv.org/abs/1807.03819v3","url_pdf":"http://arxiv.org/pdf/1807.03819v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"universal-transformers","repo_url":"https://github.com/tensorflow/tensor2tensor","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"universal-transformers","repo_url":"https://github.com/MostafaDehghani/bAbI-T2T","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":{"status":"ok","spdx":"Apache-2.0"}},{"paper_slug":"universal-transformers","repo_url":"https://github.com/andreamad8/Universal-Transformer-Pytorch","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"universal-transformers","repo_url":"https://github.com/cfiken/universal_transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"universal-transformers","repo_url":"https://github.com/kpot/keras-transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"universal-transformers","repo_url":"https://github.com/moon23k/Transformer_Archs","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":null},{"paper_slug":"universal-transformers","repo_url":"https://github.com/sidney1505/arc_maml_transformer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"tf","reach":null},{"paper_slug":"universal-transformers","repo_url":"https://github.com/ypeleg/MinimalIsAllYouNeed","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"inductive-bias","task_name":"Inductive Bias"},{"task_slug":"lambada","task_name":"LAMBADA"},{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"},{"task_slug":"learning-to-execute","task_name":"Learning to Execute"},{"task_slug":"machine-translation","task_name":"Machine Translation"},{"task_slug":"translation","task_name":"Translation"}],"methods":[{"method_slug":"absolute-position-encodings","method_name":"Absolute Position Encodings"},{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"label-smoothing","method_name":"Label Smoothing"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"position-wise-feed-forward-layer","method_name":"Position-Wise Feed-Forward Layer"},{"method_slug":"relu","method_name":"ReLU"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"transformer","method_name":"Transformer"},{"method_slug":"universal-transformer","method_name":"Universal Transformer"}],"datasets_introduced":[],"methods_introduced":[{"slug":"universal-transformer","name":"Universal Transformer","full_name":"Universal Transformer"}],"results":[{"leaderboard":"/sota/language-modelling-on-lambada","task":"Language Modelling","dataset":"LAMBADA","model":"Universal Transformer (w/ dynamic halting)","rank_in_archive_order":30,"of":37,"metrics":{"Accuracy":"56.25"},"uses_additional_data":false},{"leaderboard":"/sota/machine-translation-on-wmt2014-english-german","task":"Machine Translation","dataset":"WMT2014 English-German","model":"universal transformer base","rank_in_archive_order":37,"of":91,"metrics":{"BLEU score":"28.9"},"uses_additional_data":false}],"syntology":{"syntology_url":null,"atlas_url":"https://app.syntology.ai/?focus=1807.03819","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1807.03819"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ypeleg/MinimalIsAllYouNeed","reach":{"status":"ok"}},{"provenance":"deterministic:regex_extraction","url":"https://github.com/tensorflow/tensor2tensor","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/cfiken/universal_transformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sidney1505/arc_maml_transformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/moon23k/Transformer_Archs","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/andreamad8/Universal-Transformer-Pytorch","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/kpot/keras-transformer","reach":null},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/MostafaDehghani/bAbI-T2T","reach":{"status":"ok","spdx":"Apache-2.0"}}],"summary":{"ran":13,"ran_honours":1,"unverified":11},"by_repo_kind":{"listed":{"samples":25,"ran":14,"repositories":5}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":24,"samples":[{"code_sha256_prefix":"df7ba9e22f4021de","entry":"ACT","repo":"cfiken/universal_transformer","repo_kind":"listed","path":"model/ut.py","file_url":"https://github.com/cfiken/universal_transformer/blob/HEAD/model/ut.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"df7ba9e22f4021de"}},{"code_sha256_prefix":"ab44c54bef771f0e","entry":"EmbeddingSharedWeights","repo":"cfiken/universal_transformer","repo_kind":"listed","path":"model/ut.py","file_url":"https://github.com/cfiken/universal_transformer/blob/HEAD/model/ut.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ab44c54bef771f0e"}},{"code_sha256_prefix":"50973ad904b644b9","entry":"Embeddings","repo":"moon23k/Transformer_Archs","repo_kind":"listed","path":"model/recurrent.py","file_url":"https://github.com/moon23k/Transformer_Archs/blob/HEAD/model/recurrent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"50973ad904b644b9"}},{"code_sha256_prefix":"81f546780efe0c63","entry":"FeedForwardNetwork","repo":"cfiken/universal_transformer","repo_kind":"listed","path":"model/ut.py","file_url":"https://github.com/cfiken/universal_transformer/blob/HEAD/model/ut.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"81f546780efe0c63"}},{"code_sha256_prefix":"8187fac026534417","entry":"LayerNormalization","repo":"sidney1505/arc_maml_transformer","repo_kind":"listed","path":"keras_transformer/transformer.py","file_url":"https://github.com/sidney1505/arc_maml_transformer/blob/HEAD/keras_transformer/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8187fac026534417"}},{"code_sha256_prefix":"8f3fd62209af26f0","entry":"LayerWrapper","repo":"cfiken/universal_transformer","repo_kind":"listed","path":"model/ut.py","file_url":"https://github.com/cfiken/universal_transformer/blob/HEAD/model/ut.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"8f3fd62209af26f0"}},{"code_sha256_prefix":"99fc6b1e929c1a86","entry":"MultiHeadSelfAttention","repo":"sidney1505/arc_maml_transformer","repo_kind":"listed","path":"keras_transformer/transformer.py","file_url":"https://github.com/sidney1505/arc_maml_transformer/blob/HEAD/keras_transformer/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"99fc6b1e929c1a86"}},{"code_sha256_prefix":"97703c2875660e0b","entry":"MultiheadAttention","repo":"cfiken/universal_transformer","repo_kind":"listed","path":"model/ut.py","file_url":"https://github.com/cfiken/universal_transformer/blob/HEAD/model/ut.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"97703c2875660e0b"}},{"code_sha256_prefix":"55f808c7c4cab374","entry":"PositionalEncoding","repo":"moon23k/Transformer_Archs","repo_kind":"listed","path":"model/recurrent.py","file_url":"https://github.com/moon23k/Transformer_Archs/blob/HEAD/model/recurrent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"55f808c7c4cab374"}},{"code_sha256_prefix":"72219f63ad9d256a","entry":"SelfAttention","repo":"cfiken/universal_transformer","repo_kind":"listed","path":"model/ut.py","file_url":"https://github.com/cfiken/universal_transformer/blob/HEAD/model/ut.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"72219f63ad9d256a"}},{"code_sha256_prefix":"ac79741037ef4e2b","entry":"StandardDecoder","repo":"moon23k/Transformer_Archs","repo_kind":"listed","path":"model/recurrent.py","file_url":"https://github.com/moon23k/Transformer_Archs/blob/HEAD/model/recurrent.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":"invariant","behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"ac79741037ef4e2b"}},{"code_sha256_prefix":"d297901c2cb19187","entry":"TransformerTransition","repo":"sidney1505/arc_maml_transformer","repo_kind":"listed","path":"keras_transformer/transformer.py","file_url":"https://github.com/sidney1505/arc_maml_transformer/blob/HEAD/keras_transformer/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d297901c2cb19187"}},{"code_sha256_prefix":"1eb7bb2c7019e81d","entry":"_BaseMultiHeadAttention","repo":"sidney1505/arc_maml_transformer","repo_kind":"listed","path":"keras_transformer/transformer.py","file_url":"https://github.com/sidney1505/arc_maml_transformer/blob/HEAD/keras_transformer/transformer.py","link_basis":"first_harvest_node","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1eb7bb2c7019e81d"}},{"code_sha256_prefix":"58801803f9aacb41","entry":"generate_signal","repo":"moon23k/Transformer_Archs","repo_kind":"listed","path":"model/recurrent.py","file_url":"https://github.com/moon23k/Transformer_Archs/blob/HEAD/model/recurrent.py","link_basis":"first_harvest_node","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"58801803f9aacb41"}},{"code_sha256_prefix":"48da21be9c4b565a","entry":"ACT_basic","repo":"andreamad8/Universal-Transformer-Pytorch","repo_kind":"listed","path":"models/UTransformer.py","file_url":"https://github.com/andreamad8/Universal-Transformer-Pytorch/blob/HEAD/models/UTransformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"48da21be9c4b565a"}},{"code_sha256_prefix":"785dd04c3d9161a4","entry":"DecoderStack","repo":"cfiken/universal_transformer","repo_kind":"listed","path":"model/ut.py","file_url":"https://github.com/cfiken/universal_transformer/blob/HEAD/model/ut.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"785dd04c3d9161a4"}},{"code_sha256_prefix":"9d2c61e66b4346a7","entry":"EncoderStack","repo":"cfiken/universal_transformer","repo_kind":"listed","path":"model/ut.py","file_url":"https://github.com/cfiken/universal_transformer/blob/HEAD/model/ut.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"9d2c61e66b4346a7"}},{"code_sha256_prefix":"d65227646941963e","entry":"LayerNormalization","repo":"cfiken/universal_transformer","repo_kind":"listed","path":"model/ut.py","file_url":"https://github.com/cfiken/universal_transformer/blob/HEAD/model/ut.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d65227646941963e"}},{"code_sha256_prefix":"6dd3fd72476b96f4","entry":"ModelBase","repo":"moon23k/Transformer_Archs","repo_kind":"listed","path":"model/recurrent.py","file_url":"https://github.com/moon23k/Transformer_Archs/blob/HEAD/model/recurrent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"6dd3fd72476b96f4"}},{"code_sha256_prefix":"d939293e6a574006","entry":"RecurrentDecoder","repo":"moon23k/Transformer_Archs","repo_kind":"listed","path":"model/recurrent.py","file_url":"https://github.com/moon23k/Transformer_Archs/blob/HEAD/model/recurrent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"d939293e6a574006"}},{"code_sha256_prefix":"1e5ecdd2d7834bc5","entry":"RecurrentEncoder","repo":"moon23k/Transformer_Archs","repo_kind":"listed","path":"model/recurrent.py","file_url":"https://github.com/moon23k/Transformer_Archs/blob/HEAD/model/recurrent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"1e5ecdd2d7834bc5"}},{"code_sha256_prefix":"848001379d12e45b","entry":"RecurrentTransformer","repo":"moon23k/Transformer_Archs","repo_kind":"listed","path":"model/recurrent.py","file_url":"https://github.com/moon23k/Transformer_Archs/blob/HEAD/model/recurrent.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"848001379d12e45b"}},{"code_sha256_prefix":"bfa58c8b92ee3953","entry":"TransformerBlock","repo":"sidney1505/arc_maml_transformer","repo_kind":"listed","path":"keras_transformer/transformer.py","file_url":"https://github.com/sidney1505/arc_maml_transformer/blob/HEAD/keras_transformer/transformer.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"bfa58c8b92ee3953"}},{"code_sha256_prefix":"b3026a17c1914957","entry":"UniversalTransformer","repo":"cfiken/universal_transformer","repo_kind":"listed","path":"model/ut.py","file_url":"https://github.com/cfiken/universal_transformer/blob/HEAD/model/ut.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"NONE","inline_ok":false,"mcp_get_code":{"code_sha256":"b3026a17c1914957"}},{"code_sha256_prefix":"82a2fb5195404bdd","entry":"log_decode_results","repo":"MostafaDehghani/bAbI-T2T","repo_kind":"listed","path":"t2t_usr_dir/babi_transformer_decoder.py","file_url":"https://github.com/MostafaDehghani/bAbI-T2T/blob/HEAD/t2t_usr_dir/babi_transformer_decoder.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"Apache-2.0","inline_ok":true,"mcp_get_code":{"code_sha256":"82a2fb5195404bdd"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}