{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/transformer-language-models-without","title":"Transformer Language Models without Positional Encodings Still Learn Positional Information","arxiv_id":"2203.16634","date":"2022-03-30","proceeding":null,"authors":["Adi Haviv","Ori Ram","Ofir Press","Peter Izsak","Omer Levy"],"abstract":"Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings. However, we show that LMs without any explicit positional encoding are still competitive with standard models, and that this phenomenon is robust across different datasets, model sizes, and sequence lengths. Probing experiments reveal that such models acquire an implicit notion of absolute positions throughout the network, effectively compensating for the missing information. We conjecture that causal attention enables the model to infer the number of predecessors that each token can attend to, thereby approximating its absolute position. Our findings indicate that causal LMs might derive positional awareness not only from the explicit positioning mechanism, but also from the effects of the causal mask.","url_abs":"https://arxiv.org/abs/2203.16634v2","url_pdf":"https://arxiv.org/pdf/2203.16634v2.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"transformer-language-models-without","repo_url":"https://github.com/adihaviv/nopos","is_official":1,"mentioned_in_paper":1,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[{"task_slug":null,"task_name":"Position"}],"methods":[{"method_slug":"adam","method_name":"Adam"},{"method_slug":"attention","method_name":"Attention"},{"method_slug":"attention-dropout","method_name":"Attention Dropout"},{"method_slug":"bpe","method_name":"BPE"},{"method_slug":"cosine-annealing","method_name":"Cosine Annealing"},{"method_slug":"dense-connections","method_name":"Dense Connections"},{"method_slug":"dropout","method_name":"Dropout"},{"method_slug":"gpt-3","method_name":"GPT-3"},{"method_slug":"layer-normalization","method_name":"Layer Normalization"},{"method_slug":"linear-layer","method_name":"Linear Layer"},{"method_slug":"linear-warmup-with-cosine-annealing","method_name":"Linear Warmup With Cosine Annealing"},{"method_slug":"multi-head-attention","method_name":"Multi-Head Attention"},{"method_slug":"residual-connection","method_name":"Residual Connection"},{"method_slug":"softmax","method_name":"Softmax"},{"method_slug":"weight-decay","method_name":"Weight Decay"}],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=2203.16634","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2203.16634"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/adihaviv/nopos","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"unverified":11},"by_repo_kind":{"official":{"samples":11,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":11,"samples":[{"code_sha256_prefix":"4114512032c2bad7","entry":"get_experiment_params","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/dpp_analyzer.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/dpp_analyzer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"4114512032c2bad7"}},{"code_sha256_prefix":"e3c3533452d7f4a8","entry":"get_parameter_value","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/dpp_analyzer.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/dpp_analyzer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"e3c3533452d7f4a8"}},{"code_sha256_prefix":"39baa938c6139368","entry":"get_pe_from_name","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/dpp_analyzer.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/dpp_analyzer.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"39baa938c6139368"}},{"code_sha256_prefix":"230facb8e39690af","entry":"parse_line","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/probe_generator/probe_generator_exp_analyser.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/probe_generator/probe_generator_exp_analyser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"230facb8e39690af"}},{"code_sha256_prefix":"bfb204ebd31be12c","entry":"parse_line","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/shuffle_wikitext/shuffle_exp_analyser.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/shuffle_wikitext/shuffle_exp_analyser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"bfb204ebd31be12c"}},{"code_sha256_prefix":"8657a62f19b51024","entry":"read_paragraphs","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/the_pile_construction/pile_tokenizer_preprocess.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/the_pile_construction/pile_tokenizer_preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"8657a62f19b51024"}},{"code_sha256_prefix":"d5c56820977ac4c2","entry":"read_pile","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/the_pile_construction/pile_dataset_creator.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/the_pile_construction/pile_dataset_creator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"d5c56820977ac4c2"}},{"code_sha256_prefix":"46241bdef17fae0a","entry":"read_results","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/probe_generator/probe_generator_exp_analyser.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/probe_generator/probe_generator_exp_analyser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"46241bdef17fae0a"}},{"code_sha256_prefix":"f973bb8b0e78c3ab","entry":"read_results","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/shuffle_wikitext/shuffle_exp_analyser.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/shuffle_wikitext/shuffle_exp_analyser.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"f973bb8b0e78c3ab"}},{"code_sha256_prefix":"25c39863f6c224b4","entry":"txt2bow","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/the_pile_construction/pile_tokenizer_preprocess.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/the_pile_construction/pile_tokenizer_preprocess.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"25c39863f6c224b4"}},{"code_sha256_prefix":"9a1f2c9eb212cd3a","entry":"write_filterd_file","repo":"adihaviv/nopos","repo_kind":"official","path":"nopos_experiments/the_pile_construction/pile_dataset_creator.py","file_url":"https://github.com/adihaviv/nopos/blob/HEAD/nopos_experiments/the_pile_construction/pile_dataset_creator.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":false,"mcp_get_code":{"code_sha256":"9a1f2c9eb212cd3a"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}