{"about":{"site":"https://codewithpapers.app","non_affiliation":"Code with Papers and Syntology are not affiliated with, endorsed by, or sponsored by Papers with Code, Meta, or the pwc-archive mirror.","licence":"CC BY-SA 4.0","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","attribution":"https://codewithpapers.app/attribution","modified":"archive material modified by Syntology; see the attribution page"},"url":"/paper/pyramidal-recurrent-unit-for-language","title":"Pyramidal Recurrent Unit for Language Modeling","arxiv_id":"1808.09029","date":"2018-08-27","proceeding":"EMNLP 2018 10","authors":["Sachin Mehta","Rik Koncel-Kedziorski","Mohammad Rastegari","Hannaneh Hajishirzi"],"abstract":"LSTMs are powerful tools for modeling contextual information, as evidenced by\ntheir success at the task of language modeling. However, modeling contexts in\nvery high dimensional space can lead to poor generalizability. We introduce the\nPyramidal Recurrent Unit (PRU), which enables learning representations in high\ndimensional space with more generalization power and fewer parameters. PRUs\nreplace the linear transformation in LSTMs with more sophisticated interactions\nincluding pyramidal and grouped linear transformations. This architecture gives\nstrong results on word-level language modeling while reducing the number of\nparameters significantly. In particular, PRU improves the perplexity of a\nrecent state-of-the-art language model Merity et al. (2018) by up to 1.3 points\nwhile learning 15-20% fewer parameters. For similar number of model parameters,\nPRU outperforms all previous RNN models that exploit different gating\nmechanisms and transformations. We provide a detailed examination of the PRU\nand its behavior on the language modeling tasks. Our code is open-source and\navailable at https://sacmehta.github.io/PRU/","url_abs":"http://arxiv.org/abs/1808.09029v1","url_pdf":"http://arxiv.org/pdf/1808.09029v1.pdf","source":{"archive":"pwc-archive (Hugging Face), CC BY-SA 4.0","snapshot":"2025-07-28","licence_url":"https://creativecommons.org/licenses/by-sa/4.0/legalcode","row_kind":"abstracts"},"code_links":[{"paper_slug":"pyramidal-recurrent-unit-for-language","repo_url":"https://github.com/sacmehta/PRU","is_official":1,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"pytorch","reach":{"status":"ok","spdx":"MIT"}},{"paper_slug":"pyramidal-recurrent-unit-for-language","repo_url":"https://github.com/ifrit98/pyramidal-recurrent-layer","is_official":0,"mentioned_in_paper":0,"mentioned_in_github":1,"framework":"none","reach":{"status":"ok"}}],"tasks":[{"task_slug":"language-modeling","task_name":"Language Modeling"},{"task_slug":"language-modelling","task_name":"Language Modelling"}],"methods":[],"datasets_introduced":[],"methods_introduced":[],"results":[],"syntology":{"atlas_url":"https://app.syntology.ai/?focus=1808.09029","mcp":{"get_harvested_code_for_paper":{"arxiv_id":"1808.09029"}},"developers":"https://syntology.ai/developers","read_at":"2026-09-24T18:15:14+00:00","read_at_is":"when the build read Syntology's graph, not when any sample ran","claim":"Per-sample execution status on synthesized fixtures; not a correctness claim about the paper. Samples come from repositories linked to the paper, official or community; repo_kind says which.","repos":[{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/sacmehta/PRU","reach":{"status":"ok","spdx":"MIT"}},{"provenance":"external:paperswithcode_snapshot_2025-07-28","url":"https://github.com/ifrit98/pyramidal-recurrent-layer","reach":{"status":"ok"}}],"summary":{"unverified":3},"by_repo_kind":{"official":{"samples":3,"ran":0,"repositories":1}},"repo_kind_vocabulary":{"official":"The archive marks this repository official for the paper","named_in_paper":"The archive records that the paper mentions this repository; it is not marked official","listed":"In the archive's code links for this paper, not marked official and not recorded as mentioned in the paper","found_in_text":"Syntology found this repository in the paper's own text; whether it is the authors' implementation is not asserted","community":"Not in the archive's code links for this paper; a community repository Syntology harvested"},"n_pointer_only_for_licence":0,"samples":[{"code_sha256_prefix":"6bd10c3dd56c2b8c","entry":"batchify","repo":"sacmehta/PRU","repo_kind":"official","path":"utils.py","file_url":"https://github.com/sacmehta/PRU/blob/HEAD/utils.py","link_basis":"harvester_set","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"6bd10c3dd56c2b8c"}},{"code_sha256_prefix":"be3874d8cf97dae0","entry":"get_batch","repo":"sacmehta/PRU","repo_kind":"official","path":"utils.py","file_url":"https://github.com/sacmehta/PRU/blob/HEAD/utils.py","link_basis":"first_harvest_node","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"be3874d8cf97dae0"}},{"code_sha256_prefix":"d98e69125467f2f0","entry":"repackage_hidden","repo":"sacmehta/PRU","repo_kind":"official","path":"utils.py","file_url":"https://github.com/sacmehta/PRU/blob/HEAD/utils.py","link_basis":"plan_row","language":"python","status":"unverified","verification_level":0,"contract_check":null,"metamorphic_tier":null,"behaviour_fingerprint":false,"licence":"MIT","inline_ok":true,"mcp_get_code":{"code_sha256":"d98e69125467f2f0"}}]},"arxiv_metadata":null,"syntology_extracted_results":null}