{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/limits-to-depth-efficiencies-of-self","title":"The Depth-to-Width Interplay in Self-Attention","arxiv_id":"2006.12467","date":"2020-06-22","proceeding":"NeurIPS 2020 12","authors":["Yoav Levine","Noam Wies","Or Sharir","Hofit Bata","Amnon Shashua"],"abstract":"Self-attention architectures, which are rapidly pushing the frontier in natural language processing, demonstrate a surprising depth-inefficient behavior: previous works indicate that increasing the internal representation (network width) is just as useful as increasing the number of self-attention layers (network depth). We theoretically predict a width-dependent transition between depth-efficiency and depth-inefficiency in self-attention. We conduct systematic empirical ablations on networks of depths 6 to 48 that clearly reveal the theoretically predicted behaviors, and provide explicit quantitative suggestions regarding the optimal depth-to-width allocation for a given self-attention network size. The race towards beyond 1-Trillion parameter language models renders informed guidelines for increasing self-attention depth and width in tandem an essential ingredient. Our guidelines elucidate the depth-to-width trade-off in self-attention networks of sizes up to the scale of GPT3 (which we project to be too deep for its size), and beyond, marking an unprecedented width of 30K as optimal for a 1-Trillion parameter network.","url_abs":"https://arxiv.org/abs/2006.12467v3","url_pdf":"https://arxiv.org/pdf/2006.12467v3.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"limits-to-depth-efficiencies-of-self","repo_url":"https://github.com/uf-hobi-informatics-lab/GatorTron","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}}],"tasks":[],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"syntology_url":"https://syntology.ai/paper/2006.12467","atlas_url":"https://app.syntology.ai/?focus=2006.12467","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"2006.12467"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-25T09:33:49+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/uf-hobi-informatics-lab/GatorTron","reach":{"status":"ok","spdx":"MIT"}}],"summary":{"ran":10,"ran_violates":1,"ran_honours":1,"ran_draft_wrong":4},"by_repo_kind":{"listed":{"samples":16,"ran":16,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"88094c3b96a13d89","entry":"calc","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/re/brat_eval_res.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/re/brat_eval_res.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"88094c3b96a13d89"}},{"code_sha256_prefix":"e390dc7cb8d9947a","entry":"d2t","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"pretraining/preprocessing/gatortron_process_data.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/pretraining/preprocessing/gatortron_process_data.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e390dc7cb8d9947a"}},{"code_sha256_prefix":"bf75673211903e14","entry":"exact_match_score","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/qa/evaluate-v1.1.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/qa/evaluate-v1.1.py","link_basis":"harvester_set","language":"python","status":"ran_violates","verification_level":1,"contract_check":"VIOLATES","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"bf75673211903e14"}},{"code_sha256_prefix":"cdf4b8c0fec3da02","entry":"f1_score","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/qa/evaluate-v1.1.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/qa/evaluate-v1.1.py","link_basis":"harvester_set","language":"python","status":"ran_honours","verification_level":1,"contract_check":"HONOURS","metamorphic_tier":"well_formed","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"cdf4b8c0fec3da02"}},{"code_sha256_prefix":"82542482c691f03a","entry":"ff","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/nemo_ner/generate_conll_compatible_eval.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/nemo_ner/generate_conll_compatible_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"82542482c691f03a"}},{"code_sha256_prefix":"aa6bc32d9c17a2f1","entry":"get_tokens","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/qa/evaluate-squad-v2.0.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/qa/evaluate-squad-v2.0.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"aa6bc32d9c17a2f1"}},{"code_sha256_prefix":"ffebb9d76088e5ad","entry":"load_bio_file_into_sents","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/nemo_ner/new_bio_eval.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/nemo_ner/new_bio_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ffebb9d76088e5ad"}},{"code_sha256_prefix":"736dd0792c0ef1ce","entry":"load_gs_labels","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/nemo_ner/ner_expr.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/nemo_ner/ner_expr.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"736dd0792c0ef1ce"}},{"code_sha256_prefix":"e91c44f5680827e2","entry":"make_qid_to_has_ans","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/qa/evaluate-squad-v2.0.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/qa/evaluate-squad-v2.0.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e91c44f5680827e2"}},{"code_sha256_prefix":"ead79428fb588ca6","entry":"measure_prf","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/re/brat_eval_res.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/re/brat_eval_res.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"ead79428fb588ca6"}},{"code_sha256_prefix":"e7e75981cb464788","entry":"normalize_answer","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/qa/evaluate-v1.1.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/qa/evaluate-v1.1.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"e7e75981cb464788"}},{"code_sha256_prefix":"6a96435eba311b08","entry":"normalize_answer","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/qa/evaluate-squad-v2.0.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/qa/evaluate-squad-v2.0.py","link_basis":"harvester_set","language":"python","status":"ran_draft_wrong","verification_level":1,"contract_check":"OUTPUT_MISDECLARED","metamorphic_tier":"deterministic","behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6a96435eba311b08"}},{"code_sha256_prefix":"7caf28c5dc24bdc1","entry":"predict_in_sent","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/nemo_ner/ner_expr.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/nemo_ner/ner_expr.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"7caf28c5dc24bdc1"}},{"code_sha256_prefix":"532d336bacaef9b6","entry":"read_from_file","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/nemo_ner/new_bio_eval.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/nemo_ner/new_bio_eval.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"532d336bacaef9b6"}},{"code_sha256_prefix":"fbf1f4b2112b56ad","entry":"read_text","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/nemo_ner/nemo2bio.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/nemo_ner/nemo2bio.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"fbf1f4b2112b56ad"}},{"code_sha256_prefix":"47e420d0a8dd35ec","entry":"t2l","repo":"uf-hobi-informatics-lab/GatorTron","repo_kind":"listed","path":"finetuning/nemo_ner/nemo2bio.py","file_url":"https://github.com/uf-hobi-informatics-lab/GatorTron/blob/HEAD/finetuning/nemo_ner/nemo2bio.py","link_basis":"harvester_set","language":"python","status":"ran","verification_level":1,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":true,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"47e420d0a8dd35ec"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}